{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "4 DL Multi-Layer CNN for DigitRecognizer TF2.0.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "b6I1adl5gBQD",
        "xeMZR7ntgBQI",
        "LT78eGccgBQN",
        "4RIGVSojmy-R",
        "RX5PuSUmvRri",
        "X9uiFJ-ogBRi",
        "u13ffZdjrqQf"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JS6jP7tpCrLr",
        "colab_type": "text"
      },
      "source": [
        "## Digit Recognizer\n",
        "Learn computer vision fundamentals with the famous MNIST dat\n",
        "\n",
        "https://www.kaggle.com/c/digit-recognizer\n",
        "\n",
        "### Competition Description\n",
        "MNIST (\"Modified National Institute of Standards and Technology\") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.\n",
        "\n",
        "In this competition, your goal is to correctly identify digits from a dataset of tens of thousands of handwritten images. We’ve curated a set of tutorial-style kernels which cover everything from regression to neural networks. We encourage you to experiment with different algorithms to learn first-hand what works well and how techniques compare.\n",
        "\n",
        "### Practice Skills\n",
        "Computer vision fundamentals including simple neural networks\n",
        "\n",
        "Classification methods such as SVM and K-nearest neighbors\n",
        "\n",
        "#### Acknowledgements \n",
        "More details about the dataset, including algorithms that have been tried on it and their levels of success, can be found at http://yann.lecun.com/exdb/mnist/index.html. The dataset is made available under a Creative Commons Attribution-Share Alike 3.0 license."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LgcAAVmXgBPm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import math\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt, matplotlib.image as mpimg\n",
        "from sklearn.model_selection import train_test_split\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OsqSbpe2XSxP",
        "colab_type": "text"
      },
      "source": [
        "# Tensorflow 2.0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uTZ1PDmaRVzr",
        "colab_type": "text"
      },
      "source": [
        "### initiate TF 2.0"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9MQQlu0BRbAD",
        "colab_type": "code",
        "outputId": "a0d9e8c5-869f-4e08-ea8f-3b34c056a7b6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 2.x selected.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EITzxKZRgBPy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import models\n",
        "from tensorflow.keras import losses,optimizers,metrics\n",
        "from tensorflow.keras import layers"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "afH41ghfRhVe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.callbacks import TensorBoard"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qlSeJoslRyHS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import time\n",
        "\n",
        "\n",
        "NAME = 'DigiRecognizer-CNN-{}'.format(int(time.time()))\n",
        "tensorboard = TensorBoard(log_dir='logs/{}'.format(NAME))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pizg4suqWF6Q",
        "colab_type": "text"
      },
      "source": [
        "### Data preparation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "90bcfefe-92cc-48c4-a928-c13d6739d11f",
        "id": "5BUM4Jl6WEvI",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "from google.colab import drive \n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/gdrive; to attempt to forcibly remount, call drive.mount(\"/content/gdrive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "yReRXqpvWEvU",
        "colab": {}
      },
      "source": [
        "labeled_images = pd.read_csv('gdrive/My Drive/dataML/train.csv')\n",
        "#labeled_images = pd.read_csv('train.csv')\n",
        "images = labeled_images.iloc[:,1:]\n",
        "labels = labeled_images.iloc[:,:1]\n",
        "train_images, test_images,train_labels, test_labels = train_test_split(images, labels, test_size=0.01)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "307b465a-b6fa-4886-8a83-d1f2506c5548",
        "id": "RsiF49FiWEvW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "print(train_images.shape)\n",
        "print(train_labels.shape)\n",
        "print(test_images.shape)\n",
        "print(test_labels.shape)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(41580, 784)\n",
            "(41580, 1)\n",
            "(420, 784)\n",
            "(420, 1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uzWF5yPyVmWD"
      },
      "source": [
        "### Helper functions for batch learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "kBSohPfOVmWG",
        "colab": {}
      },
      "source": [
        "def one_hot_encode(vec, vals=10):\n",
        "    '''\n",
        "    For use to one-hot encode the 10- possible labels\n",
        "    '''\n",
        "    n = len(vec)\n",
        "    out = np.zeros((n, vals))\n",
        "    out[range(n), vec] = 1\n",
        "    return out"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "uEIs7etSVmWI",
        "colab": {}
      },
      "source": [
        "class CifarHelper():\n",
        "    \n",
        "    def __init__(self):\n",
        "        self.i = 0\n",
        "        \n",
        "        # Intialize some empty variables for later on\n",
        "        self.training_images = None\n",
        "        self.training_labels = None\n",
        "        \n",
        "        self.test_images = None\n",
        "        self.test_labels = None\n",
        "    \n",
        "    def set_up_images(self):\n",
        "        \n",
        "        print(\"Setting Up Training Images and Labels\")\n",
        "        \n",
        "        # Vertically stacks the training images\n",
        "        self.training_images = train_images.values\n",
        "        train_len = self.training_images.shape[0]\n",
        "        \n",
        "        # Reshapes and normalizes training images\n",
        "        self.training_images = self.training_images.reshape(train_len,28,28,1)/255\n",
        "        # One hot Encodes the training labels (e.g. [0,0,0,1,0,0,0,0,0,0])\n",
        "        self.training_labels = one_hot_encode(train_labels.values.reshape(-1), 10)\n",
        "        \n",
        "        print(\"Setting Up Test Images and Labels\")\n",
        "        \n",
        "        # Vertically stacks the test images\n",
        "        self.test_images = test_images.values\n",
        "        test_len = self.test_images.shape[0]\n",
        "        \n",
        "        # Reshapes and normalizes test images\n",
        "        self.test_images = self.test_images.reshape(test_len,28,28,1)/255\n",
        "        # One hot Encodes the test labels (e.g. [0,0,0,1,0,0,0,0,0,0])\n",
        "        self.test_labels = one_hot_encode(test_labels.values.reshape(-1), 10)\n",
        "\n",
        "        \n",
        "    def next_batch(self, batch_size):\n",
        "        # Note that the 100 dimension in the reshape call is set by an assumed batch size of 100\n",
        "        x = self.training_images[self.i:self.i+batch_size]\n",
        "        y = self.training_labels[self.i:self.i+batch_size]\n",
        "        self.i = (self.i + batch_size) % len(self.training_images)\n",
        "        return x, y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "6e09bc86-342a-4eae-e2bb-bb40ceac4496",
        "id": "MVjwS77gVmWL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# Before Your tf.Session run these two lines\n",
        "ch = CifarHelper()\n",
        "ch.set_up_images()\n",
        "\n",
        "# During your session to grab the next batch use this line\n",
        "# (Just like we did for mnist.train.next_batch)\n",
        "# batch = ch.next_batch(100"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Setting Up Training Images and Labels\n",
            "Setting Up Test Images and Labels\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LT78eGccgBQN",
        "colab_type": "text"
      },
      "source": [
        "### Creating the Model Class\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5SfP2N8eNkWk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class CNNModel(tf.keras.Model):\n",
        "  def __init__(self):\n",
        "    super().__init__()\n",
        "    # self.Dense1 = layers.Dense(units=10, activation='relu')\n",
        "    self.conv1 = layers.Conv2D(filters=12, kernel_size=(6,6), strides=(1,1), padding='same', activation='relu')\n",
        "    self.conv2 = layers.Conv2D(filters=24, kernel_size=(5,5), strides=(2,2), padding='same', activation='relu')\n",
        "    self.conv3 = layers.Conv2D(filters=48, kernel_size=(4,4), strides=(2,2), padding='same', activation='relu')\n",
        "    self.flatten = layers.Flatten()\n",
        "    self.dense = layers.Dense(units=10,activation='softmax')\n",
        "\n",
        "  def call(self, inputs):\n",
        "    x = self.conv1(inputs)\n",
        "    x = self.conv2(x)\n",
        "    x = self.conv3(x)\n",
        "    x = self.flatten(x)\n",
        "    output = self.dense(x)\n",
        "    return output"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YgZkz_jhH7MC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "num_epochs = 1000\n",
        "batch_size = 50\n",
        "learning_rate = 0.001"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ol_AxqEvPZWK",
        "colab_type": "text"
      },
      "source": [
        "### Initialize the model and optimizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nViiXVGTIQaJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = CNNModel()\n",
        "optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VorbeK0aPf36",
        "colab_type": "text"
      },
      "source": [
        "### Train the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZEmd0VYKIzB6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for epoch in range(num_epochs):\n",
        "  X, y_true = ch.next_batch(batch_size)\n",
        "  with tf.GradientTape() as tape:\n",
        "    y_pred = model(X)\n",
        "    loss = tf.keras.losses.categorical_crossentropy(y_true=y_true, y_pred=y_pred)\n",
        "    loss = tf.reduce_mean(loss)\n",
        "\n",
        "  grad = tape.gradient(loss, model.variables)\n",
        "  optimizer.apply_gradients(grads_and_vars=zip(grad, model.variables))\n",
        "  \n",
        "  if epoch%100 == 0:\n",
        "    print(\"epch: {}, loss: {}\".format(epoch, loss.numpy()))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "itEZ-m4APjqu",
        "colab_type": "text"
      },
      "source": [
        "### Evaluate the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9826Ub3jPp0T",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "50f87aba-8afd-426f-a5d1-2c27e4b299ec"
      },
      "source": [
        "categorical_accuracy = tf.keras.metrics.CategoricalAccuracy()\n",
        "\n",
        "y_pred = model.predict(x = ch.test_images)\n",
        "categorical_accuracy.update_state(y_true = ch.test_labels, y_pred=y_pred)\n",
        "accuracy = categorical_accuracy.result().numpy()\n",
        "print(\"The accuracy is:{}\".format(accuracy))"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The accuracy is:0.9785714149475098\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BSTvYfkJSVuc",
        "colab_type": "text"
      },
      "source": [
        "### Train and Evaluate the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kzoE7_vvSeDM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "categorical_accuracy = tf.keras.metrics.CategoricalAccuracy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hgut8gggSZNk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "outputId": "8c5ae24a-4750-40dd-a7be-310388acc9f1"
      },
      "source": [
        "for epoch in range(num_epochs):\n",
        "  X, y_true = ch.next_batch(batch_size)\n",
        "  with tf.GradientTape() as tape:\n",
        "    y_pred = model(X)\n",
        "    loss = tf.keras.losses.categorical_crossentropy(y_true=y_true, y_pred=y_pred)\n",
        "    loss = tf.reduce_mean(loss)\n",
        "\n",
        "  grad = tape.gradient(loss, model.variables)\n",
        "  optimizer.apply_gradients(grads_and_vars=zip(grad, model.variables))\n",
        "  \n",
        "  y_test_pred = model.predict(x = ch.test_images)\n",
        "  categorical_accuracy.update_state(y_true = ch.test_labels, y_pred=y_test_pred)\n",
        "  accuracy = categorical_accuracy.result().numpy()\n",
        "\n",
        "  if epoch%100 == 0:\n",
        "    print(\"epch: {}, loss: {}, accuracy: {}\".format(epoch, loss.numpy(),accuracy))"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Entity <bound method CNNModel.call of <__main__.CNNModel object at 0x7fbf5019dfd0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
            "WARNING: Entity <bound method CNNModel.call of <__main__.CNNModel object at 0x7fbf5019dfd0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
            "epch: 0, loss: 0.24360981583595276, accuracy: 0.9119047522544861\n",
            "epch: 100, loss: 0.17657403647899628, accuracy: 0.9417491555213928\n",
            "epch: 200, loss: 0.22644303739070892, accuracy: 0.9487680792808533\n",
            "epch: 300, loss: 0.0930221825838089, accuracy: 0.9535674452781677\n",
            "epch: 400, loss: 0.03676765784621239, accuracy: 0.9576594233512878\n",
            "epch: 500, loss: 0.03941649571061134, accuracy: 0.9618382453918457\n",
            "epch: 600, loss: 0.005630108527839184, accuracy: 0.9642064571380615\n",
            "epch: 700, loss: 0.05734631419181824, accuracy: 0.9666293263435364\n",
            "epch: 800, loss: 0.07206948101520538, accuracy: 0.9687057733535767\n",
            "epch: 900, loss: 0.029876455664634705, accuracy: 0.9701046347618103\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rf0cqr6kRXiu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "1eaf9309-8cda-44e1-b50d-5c212b14b053"
      },
      "source": [
        "printcategorical_accuracy.result().numpy()"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9785714"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_qetEX7AgBQY",
        "colab_type": "text"
      },
      "source": [
        "# Tensorflow 1.0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4keUL7d0gBQZ",
        "colab_type": "text"
      },
      "source": [
        "### Helper functions for batch learning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "czdbjPfcgBQd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def one_hot_encode(vec, vals=10):\n",
        "    '''\n",
        "    For use to one-hot encode the 10- possible labels\n",
        "    '''\n",
        "    n = len(vec)\n",
        "    out = np.zeros((n, vals))\n",
        "    out[range(n), vec] = 1\n",
        "    return out"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Afc2XmK2gBQh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class CifarHelper():\n",
        "    \n",
        "    def __init__(self):\n",
        "        self.i = 0\n",
        "        \n",
        "        # Intialize some empty variables for later on\n",
        "        self.training_images = None\n",
        "        self.training_labels = None\n",
        "        \n",
        "        self.test_images = None\n",
        "        self.test_labels = None\n",
        "    \n",
        "    def set_up_images(self):\n",
        "        \n",
        "        print(\"Setting Up Training Images and Labels\")\n",
        "        \n",
        "        # Vertically stacks the training images\n",
        "        self.training_images = train_images.as_matrix()\n",
        "        train_len = self.training_images.shape[0]\n",
        "        \n",
        "        # Reshapes and normalizes training images\n",
        "        self.training_images = self.training_images.reshape(train_len,28,28,1)/255\n",
        "        # One hot Encodes the training labels (e.g. [0,0,0,1,0,0,0,0,0,0])\n",
        "        self.training_labels = one_hot_encode(train_labels.as_matrix().reshape(-1), 10)\n",
        "        \n",
        "        print(\"Setting Up Test Images and Labels\")\n",
        "        \n",
        "        # Vertically stacks the test images\n",
        "        self.test_images = test_images.as_matrix()\n",
        "        test_len = self.test_images.shape[0]\n",
        "        \n",
        "        # Reshapes and normalizes test images\n",
        "        self.test_images = self.test_images.reshape(test_len,28,28,1)/255\n",
        "        # One hot Encodes the test labels (e.g. [0,0,0,1,0,0,0,0,0,0])\n",
        "        self.test_labels = one_hot_encode(test_labels.as_matrix().reshape(-1), 10)\n",
        "\n",
        "        \n",
        "    def next_batch(self, batch_size):\n",
        "        # Note that the 100 dimension in the reshape call is set by an assumed batch size of 100\n",
        "        x = self.training_images[self.i:self.i+batch_size]\n",
        "        y = self.training_labels[self.i:self.i+batch_size]\n",
        "        self.i = (self.i + batch_size) % len(self.training_images)\n",
        "        return x, y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1qAiPT1-gBQp",
        "colab_type": "code",
        "outputId": "579a01c2-db46-4e51-8279-eb66d5844e35",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# Before Your tf.Session run these two lines\n",
        "ch = CifarHelper()\n",
        "ch.set_up_images()\n",
        "\n",
        "# During your session to grab the next batch use this line\n",
        "# (Just like we did for mnist.train.next_batch)\n",
        "# batch = ch.next_batch(100)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Setting Up Training Images and Labels\n",
            "Setting Up Test Images and Labels\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QUkUv8sKgBQw",
        "colab_type": "text"
      },
      "source": [
        "## Creating the Model\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l1XSMzIpgBQx",
        "colab_type": "text"
      },
      "source": [
        "** Create 2 placeholders, x and y_true. Their shapes should be: **\n",
        "\n",
        "* X shape = [None,28,28,1]\n",
        "* Y_true shape = [None,10]\n",
        "\n",
        "** Create three more placeholders \n",
        "* lr: learning rate\n",
        "* step：for learning rate decay\n",
        "* drop_rate"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Y_4DDQvgBQz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X = tf.placeholder(tf.float32, shape=[None,28,28,1])\n",
        "Y_true = tf.placeholder(tf.float32, shape=[None,10])\n",
        "\n",
        "lr = tf.placeholder(tf.float32)\n",
        "step = tf.placeholder(tf.int32)\n",
        "drop_rate = tf.placeholder(tf.float32)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5egQVsS4NhxW",
        "colab_type": "text"
      },
      "source": [
        "### Initialize Weights and bias\n",
        " neural network structure for this sample:\n",
        "\n",
        "X [batch, 28, 28, 1]\n",
        " \n",
        "Layer 1:  conv. layer 6x6x1=>6, stride 1    W1 [6, 6, 1, 6] ,       B1 [6]\n",
        "\n",
        "Y1 [batch, 28, 28, 6]\n",
        "\n",
        "Layer 2: conv. layer 5x5x6=>12, stride 2       W2 [5, 5, 6, 12] ,       B2 [12]\n",
        "\n",
        "Y2 [batch, 14, 14, 12]\n",
        "\n",
        "Layer 3: conv. layer 4x4x12=>24, stride 2      W3 [4, 4, 12, 24]  ,     B3 [24]\n",
        "\n",
        "Y3 [batch, 7, 7, 24] => reshaped to YY [batch, 7*7*24]\n",
        "\n",
        "Layer 4: fully connected layer (relu+dropout), W4 [7*7*24, 200]       B4 [200]\n",
        "\n",
        "Y4 [batch, 200]\n",
        "\n",
        "Layer 5: fully connected layer (softmax)      W5 [200, 10]           B5 [10]\n",
        "\n",
        "Y [batch, 10]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UD6cWs60OnGn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# three convolutional layers with their channel counts, and a\n",
        "# fully connected layer (the last layer has 10 softmax neurons)\n",
        "K = 12  # first convolutional layer output depth\n",
        "L = 24  # second convolutional layer output depth\n",
        "M = 48  # third convolutional layer\n",
        "N = 200  # fully connected layer"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xiVQdlDsN1W-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "W1 = tf.Variable(tf.truncated_normal([6,6,1,K], stddev=0.1)) \n",
        "B1 = tf.Variable(tf.ones([K])/10)\n",
        "\n",
        "W2 = tf.Variable(tf.truncated_normal([5,5,K,L], stddev=0.1))\n",
        "B2 = tf.Variable(tf.ones([L])/10)\n",
        "\n",
        "W3 = tf.Variable(tf.truncated_normal([4,4,L,M], stddev=0.1))\n",
        "B3 = tf.Variable(tf.ones([M])/10)\n",
        "\n",
        "W4 = tf.Variable(tf.truncated_normal([7*7*M,N], stddev=0.1))\n",
        "B4 = tf.Variable(tf.ones([N])/10)\n",
        "\n",
        "W5 = tf.Variable(tf.truncated_normal([N, 10], stddev=0.1))\n",
        "B5 = tf.Variable(tf.zeros([10]))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u4RDJq8pO_Og",
        "colab_type": "text"
      },
      "source": [
        "### layers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DM5_O098O4Di",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "Y1 = tf.nn.relu(tf.nn.conv2d(X, W1, strides = [1,1,1,1], padding='SAME') + B1)\n",
        "\n",
        "Y2 = tf.nn.relu(tf.nn.conv2d(Y1,W2, strides = [1,2,2,1], padding='SAME') + B2)\n",
        "\n",
        "Y3 = tf.nn.relu(tf.nn.conv2d(Y2,W3, strides = [1,2,2,1], padding='SAME') + B3)\n",
        "\n",
        "#flat the inputs for the fully connected nn\n",
        "YY3 = tf.reshape(Y3, shape = (-1,7*7*M))\n",
        "                \n",
        "\n",
        "Y4 = tf.nn.relu(tf.matmul(YY3, W4) + B4)\n",
        "Y4d = tf.nn.dropout(Y4,rate = drop_rate)\n",
        "\n",
        "Ylogits = tf.matmul(Y4d, W5) + B5\n",
        "Y = tf.nn.softmax(Ylogits)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tT4TvNz-gBRI",
        "colab_type": "text"
      },
      "source": [
        "### Loss Function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wkqQ5GurgBRJ",
        "colab_type": "code",
        "outputId": "9145b309-a37b-4f3a-9b69-0b99eec766fc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "source": [
        "#cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=Y_true,logits=Ylogits))\n",
        "cross_entropy = tf.losses.softmax_cross_entropy(onehot_labels = Y_true, logits = Ylogits)\n",
        "#cross_entropy = -tf.reduce_mean(y_true * tf.log(Ylogits)) * 1000.0 "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.cast instead.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jmnEUVWxgBRM",
        "colab_type": "text"
      },
      "source": [
        "### Optimizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MoyIlzCagBRN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "lr = 0.0001 + tf.train.exponential_decay(learning_rate = 0.003, \n",
        "                                         global_step = step,\n",
        "                                         decay_steps = 2000,\n",
        "                                         decay_rate = 1/math.e\n",
        "                                        )\n",
        "\n",
        "#optimizer = tf.train.GradientDescentOptimizer(learning_rate = 0.005)\n",
        "optimizer = tf.train.AdamOptimizer(learning_rate=lr)\n",
        "train = optimizer.minimize(cross_entropy)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "47rAzeVNgBRP",
        "colab_type": "text"
      },
      "source": [
        "### Intialize Variables"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7WG1AszIgBRQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "init = tf.global_variables_initializer()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rseSYLjggBRb",
        "colab_type": "text"
      },
      "source": [
        "### Saving the Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OwdUgOG4gBRc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "saver = tf.train.Saver()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xGdHTpE0gBRe",
        "colab_type": "text"
      },
      "source": [
        "## Graph Session\n",
        "\n",
        "** Perform the training and test print outs in a Tf session and run your model! **"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pQHEYbyZgBRf",
        "colab_type": "code",
        "outputId": "cc718ec8-8ef0-42c4-b230-898ac94d6e09",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 10237
        }
      },
      "source": [
        "history = {'acc_train':list(),'acc_val':list(),\n",
        "           'loss_train':list(),'loss_val':list(),\n",
        "          'learning_rate':list()}\n",
        "with tf.Session() as sess:\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "    \n",
        "    for i in range(20000):\n",
        "        batch = ch.next_batch(100)\n",
        "        sess.run(train, feed_dict={X: batch[0], Y_true: batch[1], step: i, drop_rate: 0.25})\n",
        "        \n",
        "        # PRINT OUT A MESSAGE EVERY 100 STEPS\n",
        "        if i%100 == 0:\n",
        "            \n",
        "            # Test the Train Model\n",
        "            feed_dict_train = {X: batch[0], Y_true: batch[1], drop_rate: 0.25}\n",
        "            feed_dict_val = {X:ch.test_images, Y_true:ch.test_labels, drop_rate: 0}\n",
        "\n",
        "            matches = tf.equal(tf.argmax(Y,1),tf.argmax(Y_true,1))\n",
        "            acc = tf.reduce_mean(tf.cast(matches,tf.float32))\n",
        "            history['acc_train'].append(sess.run(acc, feed_dict = feed_dict_train))\n",
        "            history['acc_val'].append(sess.run(acc, feed_dict = feed_dict_val))\n",
        "\n",
        "            history['loss_train'].append(sess.run(cross_entropy, feed_dict = feed_dict_train))\n",
        "            history['loss_val'].append(sess.run(cross_entropy, feed_dict = feed_dict_val))\n",
        "            \n",
        "            history['learning_rate'].append(sess.run(lr, feed_dict = {step: i}))\n",
        "            print(\"Iteration {}:\\tlearning_rate={:.6f},\\tloss_train={:.6f},\\tloss_val={:.6f},\\tacc_train={:.6f},\\tacc_val={:.6f}\"\n",
        "                  .format(i,history['learning_rate'][-1],\n",
        "                          history['loss_train'][-1],\n",
        "                          history['loss_val'][-1],\n",
        "                          history['acc_train'][-1],\n",
        "                          history['acc_val'][-1]))\n",
        "            \n",
        "            print('\\n')\n",
        "        \n",
        "    saver.save(sess,'models_saving/my_model.ckpt')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Iteration 0:\tlearning_rate=0.003100,\tloss_train=3.291209,\tloss_val=3.566376,\tacc_train=0.150000,\tacc_val=0.080952\n",
            "\n",
            "\n",
            "Iteration 100:\tlearning_rate=0.002954,\tloss_train=0.273934,\tloss_val=0.113660,\tacc_train=0.950000,\tacc_val=0.973810\n",
            "\n",
            "\n",
            "Iteration 200:\tlearning_rate=0.002815,\tloss_train=0.131737,\tloss_val=0.094597,\tacc_train=0.980000,\tacc_val=0.969048\n",
            "\n",
            "\n",
            "Iteration 300:\tlearning_rate=0.002682,\tloss_train=0.058850,\tloss_val=0.089490,\tacc_train=0.980000,\tacc_val=0.971429\n",
            "\n",
            "\n",
            "Iteration 400:\tlearning_rate=0.002556,\tloss_train=0.052930,\tloss_val=0.051649,\tacc_train=0.990000,\tacc_val=0.983333\n",
            "\n",
            "\n",
            "Iteration 500:\tlearning_rate=0.002436,\tloss_train=0.108525,\tloss_val=0.043813,\tacc_train=0.970000,\tacc_val=0.980952\n",
            "\n",
            "\n",
            "Iteration 600:\tlearning_rate=0.002322,\tloss_train=0.077652,\tloss_val=0.043063,\tacc_train=0.980000,\tacc_val=0.983333\n",
            "\n",
            "\n",
            "Iteration 700:\tlearning_rate=0.002214,\tloss_train=0.062273,\tloss_val=0.050851,\tacc_train=0.990000,\tacc_val=0.983333\n",
            "\n",
            "\n",
            "Iteration 800:\tlearning_rate=0.002111,\tloss_train=0.055581,\tloss_val=0.016286,\tacc_train=1.000000,\tacc_val=0.995238\n",
            "\n",
            "\n",
            "Iteration 900:\tlearning_rate=0.002013,\tloss_train=0.063747,\tloss_val=0.026281,\tacc_train=0.990000,\tacc_val=0.985714\n",
            "\n",
            "\n",
            "Iteration 1000:\tlearning_rate=0.001920,\tloss_train=0.037742,\tloss_val=0.045727,\tacc_train=0.990000,\tacc_val=0.985714\n",
            "\n",
            "\n",
            "Iteration 1100:\tlearning_rate=0.001831,\tloss_train=0.020697,\tloss_val=0.022253,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 1200:\tlearning_rate=0.001746,\tloss_train=0.025594,\tloss_val=0.020167,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 1300:\tlearning_rate=0.001666,\tloss_train=0.005705,\tloss_val=0.022109,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 1400:\tlearning_rate=0.001590,\tloss_train=0.015113,\tloss_val=0.027408,\tacc_train=0.990000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 1500:\tlearning_rate=0.001517,\tloss_train=0.072400,\tloss_val=0.028506,\tacc_train=0.980000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 1600:\tlearning_rate=0.001448,\tloss_train=0.023121,\tloss_val=0.055556,\tacc_train=1.000000,\tacc_val=0.983333\n",
            "\n",
            "\n",
            "Iteration 1700:\tlearning_rate=0.001382,\tloss_train=0.014415,\tloss_val=0.019530,\tacc_train=1.000000,\tacc_val=0.995238\n",
            "\n",
            "\n",
            "Iteration 1800:\tlearning_rate=0.001320,\tloss_train=0.003565,\tloss_val=0.022693,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 1900:\tlearning_rate=0.001260,\tloss_train=0.015690,\tloss_val=0.055221,\tacc_train=1.000000,\tacc_val=0.983333\n",
            "\n",
            "\n",
            "Iteration 2000:\tlearning_rate=0.001204,\tloss_train=0.001746,\tloss_val=0.033166,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 2100:\tlearning_rate=0.001150,\tloss_train=0.005184,\tloss_val=0.042062,\tacc_train=0.980000,\tacc_val=0.985714\n",
            "\n",
            "\n",
            "Iteration 2200:\tlearning_rate=0.001099,\tloss_train=0.019183,\tloss_val=0.046565,\tacc_train=0.990000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 2300:\tlearning_rate=0.001050,\tloss_train=0.000312,\tloss_val=0.018573,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 2400:\tlearning_rate=0.001004,\tloss_train=0.002007,\tloss_val=0.040470,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 2500:\tlearning_rate=0.000960,\tloss_train=0.003134,\tloss_val=0.020767,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 2600:\tlearning_rate=0.000918,\tloss_train=0.002078,\tloss_val=0.027919,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 2700:\tlearning_rate=0.000878,\tloss_train=0.007485,\tloss_val=0.020951,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 2800:\tlearning_rate=0.000840,\tloss_train=0.003849,\tloss_val=0.022948,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 2900:\tlearning_rate=0.000804,\tloss_train=0.005030,\tloss_val=0.014846,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 3000:\tlearning_rate=0.000769,\tloss_train=0.000572,\tloss_val=0.021942,\tacc_train=1.000000,\tacc_val=0.995238\n",
            "\n",
            "\n",
            "Iteration 3100:\tlearning_rate=0.000737,\tloss_train=0.000241,\tloss_val=0.014402,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 3200:\tlearning_rate=0.000706,\tloss_train=0.001775,\tloss_val=0.033364,\tacc_train=0.990000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 3300:\tlearning_rate=0.000676,\tloss_train=0.000224,\tloss_val=0.018216,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 3400:\tlearning_rate=0.000648,\tloss_train=0.002972,\tloss_val=0.012856,\tacc_train=1.000000,\tacc_val=0.995238\n",
            "\n",
            "\n",
            "Iteration 3500:\tlearning_rate=0.000621,\tloss_train=0.000103,\tloss_val=0.039060,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 3600:\tlearning_rate=0.000596,\tloss_train=0.050762,\tloss_val=0.057317,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 3700:\tlearning_rate=0.000572,\tloss_train=0.000142,\tloss_val=0.038090,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 3800:\tlearning_rate=0.000549,\tloss_train=0.000966,\tloss_val=0.033859,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 3900:\tlearning_rate=0.000527,\tloss_train=0.002678,\tloss_val=0.037520,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 4000:\tlearning_rate=0.000506,\tloss_train=0.000048,\tloss_val=0.043038,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 4100:\tlearning_rate=0.000486,\tloss_train=0.000235,\tloss_val=0.035163,\tacc_train=1.000000,\tacc_val=0.983333\n",
            "\n",
            "\n",
            "Iteration 4200:\tlearning_rate=0.000467,\tloss_train=0.001476,\tloss_val=0.041278,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 4300:\tlearning_rate=0.000449,\tloss_train=0.001006,\tloss_val=0.046137,\tacc_train=1.000000,\tacc_val=0.983333\n",
            "\n",
            "\n",
            "Iteration 4400:\tlearning_rate=0.000432,\tloss_train=0.000506,\tloss_val=0.035295,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 4500:\tlearning_rate=0.000416,\tloss_train=0.000803,\tloss_val=0.031128,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 4600:\tlearning_rate=0.000401,\tloss_train=0.002269,\tloss_val=0.037874,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 4700:\tlearning_rate=0.000386,\tloss_train=0.000586,\tloss_val=0.035575,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 4800:\tlearning_rate=0.000372,\tloss_train=0.000072,\tloss_val=0.032569,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 4900:\tlearning_rate=0.000359,\tloss_train=0.004966,\tloss_val=0.041759,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 5000:\tlearning_rate=0.000346,\tloss_train=0.000024,\tloss_val=0.023334,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 5100:\tlearning_rate=0.000334,\tloss_train=0.000289,\tloss_val=0.027374,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 5200:\tlearning_rate=0.000323,\tloss_train=0.000012,\tloss_val=0.038204,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 5300:\tlearning_rate=0.000312,\tloss_train=0.003487,\tloss_val=0.037891,\tacc_train=1.000000,\tacc_val=0.985714\n",
            "\n",
            "\n",
            "Iteration 5400:\tlearning_rate=0.000302,\tloss_train=0.000134,\tloss_val=0.031178,\tacc_train=1.000000,\tacc_val=0.985714\n",
            "\n",
            "\n",
            "Iteration 5500:\tlearning_rate=0.000292,\tloss_train=0.026467,\tloss_val=0.026724,\tacc_train=0.990000,\tacc_val=0.995238\n",
            "\n",
            "\n",
            "Iteration 5600:\tlearning_rate=0.000282,\tloss_train=0.000355,\tloss_val=0.025337,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 5700:\tlearning_rate=0.000274,\tloss_train=0.001310,\tloss_val=0.032937,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 5800:\tlearning_rate=0.000265,\tloss_train=0.000005,\tloss_val=0.033200,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 5900:\tlearning_rate=0.000257,\tloss_train=0.000055,\tloss_val=0.043227,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 6000:\tlearning_rate=0.000249,\tloss_train=0.002238,\tloss_val=0.050428,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 6100:\tlearning_rate=0.000242,\tloss_train=0.000573,\tloss_val=0.031664,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 6200:\tlearning_rate=0.000235,\tloss_train=0.000065,\tloss_val=0.036321,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 6300:\tlearning_rate=0.000229,\tloss_train=0.000200,\tloss_val=0.028852,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 6400:\tlearning_rate=0.000222,\tloss_train=0.000258,\tloss_val=0.030592,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 6500:\tlearning_rate=0.000216,\tloss_train=0.000016,\tloss_val=0.022441,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 6600:\tlearning_rate=0.000211,\tloss_train=0.007413,\tloss_val=0.032315,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 6700:\tlearning_rate=0.000205,\tloss_train=0.001362,\tloss_val=0.034481,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 6800:\tlearning_rate=0.000200,\tloss_train=0.000106,\tloss_val=0.037938,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 6900:\tlearning_rate=0.000195,\tloss_train=0.001134,\tloss_val=0.046206,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 7000:\tlearning_rate=0.000191,\tloss_train=0.000098,\tloss_val=0.040202,\tacc_train=0.990000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 7100:\tlearning_rate=0.000186,\tloss_train=0.000029,\tloss_val=0.037919,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 7200:\tlearning_rate=0.000182,\tloss_train=0.002708,\tloss_val=0.049570,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 7300:\tlearning_rate=0.000178,\tloss_train=0.000150,\tloss_val=0.065175,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 7400:\tlearning_rate=0.000174,\tloss_train=0.000034,\tloss_val=0.036139,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 7500:\tlearning_rate=0.000171,\tloss_train=0.000393,\tloss_val=0.045847,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 7600:\tlearning_rate=0.000167,\tloss_train=0.000328,\tloss_val=0.057202,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 7700:\tlearning_rate=0.000164,\tloss_train=0.000026,\tloss_val=0.045608,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 7800:\tlearning_rate=0.000161,\tloss_train=0.001196,\tloss_val=0.047827,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 7900:\tlearning_rate=0.000158,\tloss_train=0.000038,\tloss_val=0.043284,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 8000:\tlearning_rate=0.000155,\tloss_train=0.000109,\tloss_val=0.041985,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 8100:\tlearning_rate=0.000152,\tloss_train=0.000008,\tloss_val=0.057204,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 8200:\tlearning_rate=0.000150,\tloss_train=0.000007,\tloss_val=0.061682,\tacc_train=1.000000,\tacc_val=0.985714\n",
            "\n",
            "\n",
            "Iteration 8300:\tlearning_rate=0.000147,\tloss_train=0.000008,\tloss_val=0.056300,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 8400:\tlearning_rate=0.000145,\tloss_train=0.000237,\tloss_val=0.053918,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 8500:\tlearning_rate=0.000143,\tloss_train=0.000021,\tloss_val=0.060757,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 8600:\tlearning_rate=0.000141,\tloss_train=0.000096,\tloss_val=0.056191,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 8700:\tlearning_rate=0.000139,\tloss_train=0.000090,\tloss_val=0.043130,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 8800:\tlearning_rate=0.000137,\tloss_train=0.000013,\tloss_val=0.044164,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 8900:\tlearning_rate=0.000135,\tloss_train=0.001191,\tloss_val=0.056158,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 9000:\tlearning_rate=0.000133,\tloss_train=0.002281,\tloss_val=0.063807,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 9100:\tlearning_rate=0.000132,\tloss_train=0.000023,\tloss_val=0.064389,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 9200:\tlearning_rate=0.000130,\tloss_train=0.000010,\tloss_val=0.062237,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 9300:\tlearning_rate=0.000129,\tloss_train=0.000011,\tloss_val=0.055023,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 9400:\tlearning_rate=0.000127,\tloss_train=0.001363,\tloss_val=0.060808,\tacc_train=0.990000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 9500:\tlearning_rate=0.000126,\tloss_train=0.000015,\tloss_val=0.069666,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 9600:\tlearning_rate=0.000125,\tloss_train=0.000559,\tloss_val=0.066412,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 9700:\tlearning_rate=0.000123,\tloss_train=0.000048,\tloss_val=0.049534,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 9800:\tlearning_rate=0.000122,\tloss_train=0.000039,\tloss_val=0.056987,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 9900:\tlearning_rate=0.000121,\tloss_train=0.000081,\tloss_val=0.049058,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 10000:\tlearning_rate=0.000120,\tloss_train=0.000057,\tloss_val=0.044235,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 10100:\tlearning_rate=0.000119,\tloss_train=0.000252,\tloss_val=0.047751,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 10200:\tlearning_rate=0.000118,\tloss_train=0.000002,\tloss_val=0.047568,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 10300:\tlearning_rate=0.000117,\tloss_train=0.000223,\tloss_val=0.048713,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 10400:\tlearning_rate=0.000117,\tloss_train=0.000000,\tloss_val=0.042749,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 10500:\tlearning_rate=0.000116,\tloss_train=0.000005,\tloss_val=0.038871,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 10600:\tlearning_rate=0.000115,\tloss_train=0.000074,\tloss_val=0.030597,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 10700:\tlearning_rate=0.000114,\tloss_train=0.000003,\tloss_val=0.041348,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 10800:\tlearning_rate=0.000114,\tloss_train=0.000746,\tloss_val=0.034549,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 10900:\tlearning_rate=0.000113,\tloss_train=0.000008,\tloss_val=0.032043,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 11000:\tlearning_rate=0.000112,\tloss_train=0.000000,\tloss_val=0.033110,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 11100:\tlearning_rate=0.000112,\tloss_train=0.000002,\tloss_val=0.053547,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 11200:\tlearning_rate=0.000111,\tloss_train=0.000012,\tloss_val=0.056942,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 11300:\tlearning_rate=0.000111,\tloss_train=0.000033,\tloss_val=0.058680,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 11400:\tlearning_rate=0.000110,\tloss_train=0.000013,\tloss_val=0.059468,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 11500:\tlearning_rate=0.000110,\tloss_train=0.000600,\tloss_val=0.052409,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 11600:\tlearning_rate=0.000109,\tloss_train=0.000304,\tloss_val=0.050763,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 11700:\tlearning_rate=0.000109,\tloss_train=0.000391,\tloss_val=0.043252,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 11800:\tlearning_rate=0.000108,\tloss_train=0.000051,\tloss_val=0.043646,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 11900:\tlearning_rate=0.000108,\tloss_train=0.000001,\tloss_val=0.045690,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 12000:\tlearning_rate=0.000107,\tloss_train=0.000022,\tloss_val=0.036975,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 12100:\tlearning_rate=0.000107,\tloss_train=0.001141,\tloss_val=0.028697,\tacc_train=1.000000,\tacc_val=0.995238\n",
            "\n",
            "\n",
            "Iteration 12200:\tlearning_rate=0.000107,\tloss_train=0.002605,\tloss_val=0.037994,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 12300:\tlearning_rate=0.000106,\tloss_train=0.000231,\tloss_val=0.038053,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 12400:\tlearning_rate=0.000106,\tloss_train=0.000008,\tloss_val=0.041612,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 12500:\tlearning_rate=0.000106,\tloss_train=0.000435,\tloss_val=0.046244,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 12600:\tlearning_rate=0.000106,\tloss_train=0.000046,\tloss_val=0.046227,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 12700:\tlearning_rate=0.000105,\tloss_train=0.000006,\tloss_val=0.042901,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 12800:\tlearning_rate=0.000105,\tloss_train=0.000022,\tloss_val=0.041342,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 12900:\tlearning_rate=0.000105,\tloss_train=0.000032,\tloss_val=0.041868,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 13000:\tlearning_rate=0.000105,\tloss_train=0.000060,\tloss_val=0.041549,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 13100:\tlearning_rate=0.000104,\tloss_train=0.000057,\tloss_val=0.031410,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 13200:\tlearning_rate=0.000104,\tloss_train=0.000524,\tloss_val=0.035047,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 13300:\tlearning_rate=0.000104,\tloss_train=0.000002,\tloss_val=0.036295,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 13400:\tlearning_rate=0.000104,\tloss_train=0.000502,\tloss_val=0.042062,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 13500:\tlearning_rate=0.000104,\tloss_train=0.000056,\tloss_val=0.041174,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 13600:\tlearning_rate=0.000103,\tloss_train=0.000025,\tloss_val=0.038606,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 13700:\tlearning_rate=0.000103,\tloss_train=0.000005,\tloss_val=0.042296,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 13800:\tlearning_rate=0.000103,\tloss_train=0.000003,\tloss_val=0.045483,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 13900:\tlearning_rate=0.000103,\tloss_train=0.000003,\tloss_val=0.047729,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 14000:\tlearning_rate=0.000103,\tloss_train=0.000010,\tloss_val=0.047209,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 14100:\tlearning_rate=0.000103,\tloss_train=0.000014,\tloss_val=0.045859,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 14200:\tlearning_rate=0.000102,\tloss_train=0.000199,\tloss_val=0.050122,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 14300:\tlearning_rate=0.000102,\tloss_train=0.000343,\tloss_val=0.041970,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 14400:\tlearning_rate=0.000102,\tloss_train=0.000013,\tloss_val=0.042751,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 14500:\tlearning_rate=0.000102,\tloss_train=0.000017,\tloss_val=0.047536,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 14600:\tlearning_rate=0.000102,\tloss_train=0.001420,\tloss_val=0.038829,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 14700:\tlearning_rate=0.000102,\tloss_train=0.000053,\tloss_val=0.042733,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 14800:\tlearning_rate=0.000102,\tloss_train=0.000122,\tloss_val=0.036415,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 14900:\tlearning_rate=0.000102,\tloss_train=0.000097,\tloss_val=0.034187,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 15000:\tlearning_rate=0.000102,\tloss_train=0.000019,\tloss_val=0.036098,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 15100:\tlearning_rate=0.000102,\tloss_train=0.000010,\tloss_val=0.053080,\tacc_train=1.000000,\tacc_val=0.995238\n",
            "\n",
            "\n",
            "Iteration 15200:\tlearning_rate=0.000102,\tloss_train=0.000594,\tloss_val=0.047981,\tacc_train=1.000000,\tacc_val=0.995238\n",
            "\n",
            "\n",
            "Iteration 15300:\tlearning_rate=0.000101,\tloss_train=0.000234,\tloss_val=0.046626,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 15400:\tlearning_rate=0.000101,\tloss_train=0.000383,\tloss_val=0.048023,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 15500:\tlearning_rate=0.000101,\tloss_train=0.000006,\tloss_val=0.052817,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 15600:\tlearning_rate=0.000101,\tloss_train=0.006560,\tloss_val=0.060489,\tacc_train=0.990000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 15700:\tlearning_rate=0.000101,\tloss_train=0.000015,\tloss_val=0.044697,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 15800:\tlearning_rate=0.000101,\tloss_train=0.000000,\tloss_val=0.040762,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 15900:\tlearning_rate=0.000101,\tloss_train=0.000000,\tloss_val=0.040853,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 16000:\tlearning_rate=0.000101,\tloss_train=0.000048,\tloss_val=0.040024,\tacc_train=1.000000,\tacc_val=0.995238\n",
            "\n",
            "\n",
            "Iteration 16100:\tlearning_rate=0.000101,\tloss_train=0.000001,\tloss_val=0.041087,\tacc_train=1.000000,\tacc_val=0.995238\n",
            "\n",
            "\n",
            "Iteration 16200:\tlearning_rate=0.000101,\tloss_train=0.000482,\tloss_val=0.047162,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 16300:\tlearning_rate=0.000101,\tloss_train=0.000048,\tloss_val=0.038735,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 16400:\tlearning_rate=0.000101,\tloss_train=0.000004,\tloss_val=0.040919,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 16500:\tlearning_rate=0.000101,\tloss_train=0.000111,\tloss_val=0.028003,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 16600:\tlearning_rate=0.000101,\tloss_train=0.000004,\tloss_val=0.026905,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 16700:\tlearning_rate=0.000101,\tloss_train=0.000041,\tloss_val=0.018132,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 16800:\tlearning_rate=0.000101,\tloss_train=0.000007,\tloss_val=0.019089,\tacc_train=1.000000,\tacc_val=0.995238\n",
            "\n",
            "\n",
            "Iteration 16900:\tlearning_rate=0.000101,\tloss_train=0.000041,\tloss_val=0.024353,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 17000:\tlearning_rate=0.000101,\tloss_train=0.000008,\tloss_val=0.021087,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 17100:\tlearning_rate=0.000101,\tloss_train=0.000031,\tloss_val=0.019918,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 17200:\tlearning_rate=0.000101,\tloss_train=0.000012,\tloss_val=0.021076,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 17300:\tlearning_rate=0.000101,\tloss_train=0.004532,\tloss_val=0.034122,\tacc_train=0.990000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 17400:\tlearning_rate=0.000100,\tloss_train=0.000005,\tloss_val=0.033327,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 17500:\tlearning_rate=0.000100,\tloss_train=0.000002,\tloss_val=0.035994,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 17600:\tlearning_rate=0.000100,\tloss_train=0.000011,\tloss_val=0.035315,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 17700:\tlearning_rate=0.000100,\tloss_train=0.000013,\tloss_val=0.049643,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 17800:\tlearning_rate=0.000100,\tloss_train=0.000011,\tloss_val=0.055924,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 17900:\tlearning_rate=0.000100,\tloss_train=0.000070,\tloss_val=0.053377,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 18000:\tlearning_rate=0.000100,\tloss_train=0.000005,\tloss_val=0.054083,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 18100:\tlearning_rate=0.000100,\tloss_train=0.000000,\tloss_val=0.049477,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 18200:\tlearning_rate=0.000100,\tloss_train=0.000001,\tloss_val=0.048280,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 18300:\tlearning_rate=0.000100,\tloss_train=0.000000,\tloss_val=0.050973,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 18400:\tlearning_rate=0.000100,\tloss_train=0.002151,\tloss_val=0.050878,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 18500:\tlearning_rate=0.000100,\tloss_train=0.004797,\tloss_val=0.060612,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 18600:\tlearning_rate=0.000100,\tloss_train=0.001980,\tloss_val=0.051940,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 18700:\tlearning_rate=0.000100,\tloss_train=0.000092,\tloss_val=0.037035,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 18800:\tlearning_rate=0.000100,\tloss_train=0.000000,\tloss_val=0.036092,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 18900:\tlearning_rate=0.000100,\tloss_train=0.000002,\tloss_val=0.036987,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 19000:\tlearning_rate=0.000100,\tloss_train=0.000039,\tloss_val=0.035656,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 19100:\tlearning_rate=0.000100,\tloss_train=0.022560,\tloss_val=0.030500,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 19200:\tlearning_rate=0.000100,\tloss_train=0.000002,\tloss_val=0.046482,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 19300:\tlearning_rate=0.000100,\tloss_train=0.000039,\tloss_val=0.055406,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 19400:\tlearning_rate=0.000100,\tloss_train=0.000081,\tloss_val=0.041961,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 19500:\tlearning_rate=0.000100,\tloss_train=0.000001,\tloss_val=0.042968,\tacc_train=1.000000,\tacc_val=0.988095\n",
            "\n",
            "\n",
            "Iteration 19600:\tlearning_rate=0.000100,\tloss_train=0.000026,\tloss_val=0.034487,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 19700:\tlearning_rate=0.000100,\tloss_train=0.000002,\tloss_val=0.035798,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n",
            "Iteration 19800:\tlearning_rate=0.000100,\tloss_train=0.000000,\tloss_val=0.039822,\tacc_train=1.000000,\tacc_val=0.992857\n",
            "\n",
            "\n",
            "Iteration 19900:\tlearning_rate=0.000100,\tloss_train=0.000042,\tloss_val=0.041927,\tacc_train=1.000000,\tacc_val=0.990476\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s5rC6lSIAR-P",
        "colab_type": "code",
        "outputId": "1c59d87f-9158-4a6d-8f25-151c9890c407",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "source": [
        "plt.plot(history['acc_train'],'b')\n",
        "plt.plot(history['acc_val'],'r')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f18508a9748>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHvtJREFUeJzt3XucVWWh//HPvsydAQYcbgoIBY8X\nSgNUULmLIh41b9VJKzvW+XXCXppp2ckyy/LXRfHSRc0OZR2rn5mEeYlEuSSigIgh9GggigIyXIaZ\nYS77tn5/7Av7NsOecfbMPPV9v16+nL332mt/Z7HmO888a629fZ7nISIi7vL3dgAREXlvVOQiIo5T\nkYuIOE5FLiLiOBW5iIjjgj39gnV1jV0+TaamppIDB5q7M0636avZlKtz+mou6LvZlKtzupqrtrba\n195jTo3Ig8FAb0doV1/Nplyd01dzQd/NplydU4xcThW5iIjkUpGLiDhORS4i4jgVuYiI41TkIiKO\nK+j0Q2PMBOCPwEJr7Y+yHjsL+C4QBZ6w1n6721OKiEi7jjgiN8ZUAfcAy9pZ5G7gEuAM4GxjzAnd\nF09ERI6kkBF5GzAf+Er2A8aYscB+a+2OxO0ngDnA5u4M2RWHDsFPf1rKhRdGGDcuBsCGDX4WLSol\nGo0vM3JkjGuvDVFeHr99//0lbNyY/xzPadMifOxjEQAOHoSFC8uoqzt8fn55ObS2lrebx+eDj388\nzNSpURYtKmH9+p45x7W9XLNnR7jkkghPPRXgscdKuvU158+PcN55ERYvDvKXv+TfxY60vbpTWZnH\n5z4XZvToGHfdVcqbb7Y/funJXJ3VV7N1lGvgQI9rrgkRCMCdd5ayf3+717S8J/lep69tL78frrwy\nxLx53b/uIxa5tTYCRIwx+R4eBtSl3d4DvK+j9dXUVL6nE+Jra6sLWu7mr7TSb9FdfP03F7H41XE0\nNcHHPw6+fXUEiLKHIXiU4Hll3HEHbN0KN90EI3mLq/kRK5nOUs4mTCkAS5aU8MlPQv/+sGAB/OHh\nCFECQPqOmVuIE1nPR/kdT3Iun3x8OtfdEOCH32zk33iUSaxnOTMzXqcrymiljbKsLAAefqLE8uRa\n9nADJ/7+R3xz+ZVsi44uaN0+Yqm1duQPfyjh1lvh61+NEMnaRtU0UEkzDfSnhcq8z6+iiX40dbhM\nZ61eXcq8efCTn+R/vJJDVNOYcV8bZdRTk7pdQohB7G/3NWL4qaOW3H+HbB4VtGR8bxU0M58nOJmX\n+SWf4h+Ma+e58X9LP1FqqcPDxx6GAL7UdkvXQgUNDAA8BrOPIJEjZOucEsLM4lnm8Bx+YqxgBv/L\n5aRvg61bSykrgzVP7qeUEADVNHI+j3E8W4gQ5NdcwWrOoB+NVHGIQ1TRxOGf9QHUcwFL+AB/42nO\n4hU+iJe1nfdvhrIyWPZMt36L7creP9rjI0YpIdooZ/z4EubNK7zHCuUr9IMljDHfBPamz5EbY04H\nbrDWXpS4/RlgrLX2v9tbz3u5RL+2tpq6uswftpYWeOSREj784TD9+sXve/zxIG99+ja+wbfZyliu\nnfYi+5or+dL6K7iEPwAQqh3Ob8KXcn/9R7nh56NZ92o/vnPHAN4YO4tjt60A4NCEybz+syf55e+q\nufPOUu68s5WSSAuHrr+NL/ruJDJqDAfOvpiGafOofdvS9tYOGqbMpnzb3yndtYO20e9n5G1fJHAo\nnnk3Q9nISUxnJRW0pr6HtmPG8NZN99AyfgJVr66nes0z+EIhwkOGUz/7AtrGJH6JhsNQUoIv1MaQ\nB+8mPHQEgaYGhv/oFlrGncjuz95I5eaXiNQchVdWzog7b8LvxTgw63zq515M4+TpEAzyxlbo95FL\nONtbypuM4o2rb+O4Qy/RctxJRGqOovqFZ2kdexwl777D8Hu/w6EPnkbD6Wcx7IHvE62qpv6sD1M/\n+wJK9u6m38trIBaj7dhxHJwxn7+/Xc2Dn9vAZTzMBSwhcNRAdn9jIc0nTGTIg3cx5Nf34IvF8MrK\nOHj6XOrPuojQ0KPp/9xSAk2NlO3YRvXa5fiiUaIVVWz/3i9pOPMcyrduYcDyPxGtqqZh+rlEK6qo\neuUFql9ciS8SBsALBmmaPI1YZT+qV/8Ff2t8G2/e4mfTpvjgoX9/j7PnRvCn/S4q3fUW1WuewZ9Y\nT7pDEybTfPzJBA8eiGc81JizTLq2Y8bQeMoMvGD+v3J8kTD91q2ifMdWmk6eQsu4CQT319H/ub8Q\naI1fth0rr2D/vMuIVVSlvp9+L64gcMpp1Af7MfDpxQxctpiSfXvi+/KwkbSNHEu/Dc/hi+QWddNJ\npxFoqKfiDdth9u7SOHk6rYl9ds2aADt2+DiVF5nM+naf4/l8NJ8wkcotG+L7RyBA4ykzaBv5Pkp3\n74j/+4RDPZK/M/bP/xg7r74ZYjEGrHqKQGM9jafNosL+jYrXN+EPtVL9/DJK9r1L0+TphG+9hUFn\nTcvpsUJ0dIn+ey3yY4HfWGunJm7fDOzLPiCarruL/Npry3jooVKuv76NL385xLtvhfjs7N082zCJ\nUn8EfyzKKs4kSoCZrCA84YPERo2mZPUq/PX1qfW0Ucqf/BdySexhQtNm4FVVUfbUE7TNPx9v0xZa\n3trLhn7T+GDT8xzFXkK1wylpqsfX0tJhZi8Q4NBNtxDY/gah3y5hQFsd+4cdT9kVFxKefCplTz1O\n+YOL8MVi7a6j9eLLiI4bT+WdPyQ0ey6EQ5Q9vfTwa1RU5M3hVVbi698fdu8GIDZoEKG58/A1NlL2\nxGNsZSzvY1vH+dPWHRs4ECBju3XkQP+RDGzelVEu0VHHEv7QRMr/YeHVV/M+L3zSh4geO4aypU9C\nNEr02DEEXytuCUVOmEBk/HjKy0pobYsXun/vXkqefw5fYi4uOnIU4Q9NAn/+nydfSwulq1biaz7U\n4Wt5lZVExhuCG1/Gl/j5i4wZS9uFFxMbOYqq796Cf9++DtcRGzSI8NQzIRql5LlV+Bsb4tttzJiM\n5fy7d1PywvNQVkbojGl41d07EgQonzyRA1NnAFD17ZspXZ47JA4TpO20MwkOH5QIFiA89QzCU8/A\nv/Md+t34JYLbthKeNJnoyFEE3txOyYaXUs+PnPgB2i74MOGTTqb02WX4392dsf62Nh/PPhPA82DW\n7Cjl5V7Gv2UxBF97jeDmTUdcLjZgINGRoyjZ9ArN13yJyjt/2LeKPHH/q8B5wNvA88Dl1trX2ltP\ndxR5OAwrVgTYvDnAmltX8hjn82rZRCacVk5g5UqCxH/wGn7yM0p/9SDlz68CoG36LBp+9VuoqIBw\nmJJVy/nH95/grZf2cxovcDQ78crK2L/yBWJDh1Fz7myCWzbj+f3sDQylNryL3Qxl19lXcMx914Pn\nUfb0nylZ8SwVk07mYP/BlK54luixY4mOGUvps8sInT2P0DnnAhALRdj58l6OOXVYxvcVXPciFQ/c\nh6+1lejIUYTOPY9YzSCCm16h4oF7Uzu0V1aGr60NgNCsOYSnnoGvvp7m//oCJevXUrriGcKnn4l/\n9y78b26n5bP/xeBJE6h//C+ULXmU0sf+SGDPu0C8lF7+2QrG/e2PBF/7O+FpMwluWIe/oYHQ9FkE\nN72Cr62N5s9/gdLlzxBc9yLN11yPN2gQJauWU/bnJ4kNGkxo7jl4ZeWUrFkdL71IhMiYsbwx6SKG\nnvchAq9uovInd+NrbiZy3HE0f+E6qKqitraa/X9dS9mSR/HX7SE09xyiI0fjDRhAbPgIAEpWLmfA\nJz4Knkdo9lzazr8QX3196nWiY8bSds58vMQvGN/Bg/Hyb2sldPa5xIYMTW3jtjY4eNDHkCG5u59X\nXU3s6GMy9rEk34H9+N99F6+0lNiYsfGDHR1paSHw5vYOF4mOGg2Vlfj27sW/tw6vvJzY6GMPr7up\nicDbOzK+n/CZMxjw2iZadu2hbd55hM+YBsFg6pvzNTTg1dbmfT3fgf14JaWk/mTtZhnbzPPwv7EN\nX+jw6LmxEUJHDWPgmA6mISIRfE2NeAMPL+PfvQtffT1ev37Ejhl5xBz19RCN+hg82MvNVQyRSPzn\nc83zEAgQmno63uCjKFm5nOj44widOR1KS4mOGQulpfj278Or7k/tiEE9X+TGmEnA7cCxQBh4B1gC\nvGGtfdQYMx34XmLxR6y1P+xofd1R5L/+dQnXXVdOGa1s9p3IaG87AAFirGMS0aOGMuHfj6P5pvif\nPIEtm8HvJ3rc8WT8TQ3EYnDZZRW8sqqJ5877FiPPn0DbxZcB4N+2lcp7FtL6iSt54OXT+MmN73LK\nxcP58b25v+WLttNEIlT8/D78b++g+bovU7ZkMYF/vM6hr91M6ihtBzJyRaMEXrP4ImEi7xsHld0z\n/9wVhW4vX10dVJTj9ev+kWQ+Rf/hfw/6ajbl6pyu5uqWEXl3ea9FHpozl+0bGvjkgbu4Y/KvOX3d\nj/n7uVcz7clvUEqIcO1wli9vpra28Jc5eBD++tcg8+dH2h1wRSLwxBNB5s6NUFGRP9s/005TbMrV\neX01m3J1TjGKvMffj/y9Kn3macYDa5gK6yA6bDgDFt6I9+Jgdu7z89t7OlfiAAMGwHnndXw0PxiE\nCy7o3iP+IiLdwbkij1b1J3CogeeHXsAH/+MkWj92OcFB/bn33lb27vUxe3a0tyOKiPQo94o8EuNl\nJrLs6t/x/v9zeK56xgwVuIj8a3LuTbO8SIwYfqZMUXGLiICLRR6Lgd/PiSe2f961iMi/EueK3OfF\nKCnzpU6hFRH5V+dckfuJ4fmciy0iUjTONaKKXEQkk3ONGFCRi4hkcKsRE1ehHultVEVE/pW41YiJ\ndwjUiFxE5DC3GlFFLiKSw61GVJGLiORwqhG9qIpcRCSbU40Yi6jIRUSyOdWI0bCKXEQkm1ONmBqR\n+52KLSJSVE41Yiwcf8dDjchFRA5zqhGTI3JU5CIiKU41YrLIY5paERFJcaoRddaKiEgupxpRUysi\nIrmcakSNyEVEcjnViDr9UEQkl1ONmLxEX1MrIiKHOdWIGpGLiORyqhF1sFNEJJdTjaiDnSIiuZxq\nxNQcud/Xu0FERPoQR4vcqdgiIkXlVCPqbWxFRHI51YipTwjSiFxEJMWpRtTUiohILqcaUacfiojk\nChaykDFmITAF8IBrrLVr0x5bAFwBRIF11tprixEUIKYRuYhIjiM2ojFmBjDOWjsVuAq4O+2x/sAN\nwDRr7ZnACcaYKcUK6+k8chGRHIU04hxgMYC1dgtQkyhwgFDiv37GmCBQCewvRlBImyMPqMhFRJIK\nmVoZBqxPu12XuK/BWttqjLkF2Aa0AL+11r7W0cpqaioJBgNdCvtuYkReUlpCbW11l9ZRTH0xEyhX\nZ/XVXNB3sylX53R3roLmyLOkLqtMjMz/GxgPNADPGGNOstZubO/JBw40d+El45Ij8nA0Rl1dY5fX\nUwy1tdV9LhMoV2f11VzQd7MpV+d0NVdH5V/IHMVO4iPwpBHArsTXxwPbrLV7rbUhYBUwqdMJC6TT\nD0VEchXSiEuBSwGMMROBndba5K+T7cDxxpiKxO3JwOvdHTIpdfqhilxEJOWIUyvW2tXGmPXGmNVA\nDFhgjLkSOGitfdQY8wPgWWNMBFhtrV1VrLAakYuI5Cpojtxae2PWXRvTHrsPuK87Q7UrGo3/X0Uu\nIpLiVCOm3msl0LWzXkRE/hk5VeTJOXKfRuQiIilONWJqRK4rO0VEUpxqRC+mTwgSEcnmVJGTGJH7\ndIm+iEiKU42o88hFRHI51YiHp1acii0iUlRuNaLe/VBEJIdTjZj8YAmdfigicphbjahL9EVEcjjV\niHqvFRGRXG41Yix5+qHOIxcRSXKqyDUiFxHJ5VQjqshFRHK51YgxXdkpIpLNqUb0dIm+iEgOpxpR\nUysiIrncakRNrYiI5HCrEWO6RF9EJJtTjejpEn0RkRxONaKnC4JERHI4VeR6rxURkVxuNWJiRO4P\nuhVbRKSY3GrEaDT+f43IRURS3GpEnbUiIpLDrUbU1IqISA6nGlGX6IuI5HKrEfXhyyIiOdxqRF2i\nLyKSw61G1AVBIiI5HC1yt2KLiBSTW42oIhcRyeFWI6rIRURyuNWIKnIRkRzBQhYyxiwEpgAecI21\ndm3aYyOB3wClwEvW2s8VIyigIhcRyeOIjWiMmQGMs9ZOBa4C7s5a5HbgdmvtqUDUGDOq+2MmeCpy\nEZFshTTiHGAxgLV2C1BjjOkPYIzxA9OAJYnHF1hr3ypSVny6RF9EJEchUyvDgPVpt+sS9zUAtUAj\nsNAYMxFYZa39akcrq6mpJBgMdCls8oMlBg6qora2ukvrKKa+mAmUq7P6ai7ou9mUq3O6O1dBc+RZ\nfFlfHw3cBWwHHjfGnGetfby9Jx840NyFl0y8WKLIG5paqatr7PJ6iqG2trrPZQLl6qy+mgv6bjbl\n6pyu5uqo/AuZo9hJfASeNALYlfh6L/CmtXartTYKLANO7HTCQmmOXEQkRyGNuBS4FCAxfbLTWtsI\nYK2NANuMMeMSy04CbDGCQvocuS7RFxFJOuLUirV2tTFmvTFmNRADFhhjrgQOWmsfBa4FfpE48Pk3\n4LGipdXBThGRHAXNkVtrb8y6a2PaY/8AzuzOUO3S1IqISA6nGtGnC4JERHK41YiaWhERyeFUI/o0\ntSIiksOtRvR01oqISDanitwXiwKaWhERSedUI+q9VkREcjnViMk5cgJde68WEZF/Rk4VuebIRURy\nOVXkmloREcnlVCP6PBW5iEg2pxpRRS4iksutRkwe7PS7FVtEpJicakRd2SkiksupRvRpRC4iksOp\nRlSRi4jkcqoRk6cfqshFRA5zqhE1IhcRyeVUI/pIFLlPV3aKiCS5VeRejKhbkUVEis6pVvR5MWJu\nRRYRKTqnWlFFLiKSy6lWVJGLiORyqhX9XgzPrcgiIkXnVCv6vBgxn1ORRUSKzqlW9KGpFRGRbE61\noqZWRERyOdWKmloREcnlVCtqakVEJJdTrRjwongakYuIZHCqFTUiFxHJ5VQrxufIA70dQ0SkT3Gq\nyP3orBURkWxOtaJfZ62IiORwqhV9GpGLiOQIFrKQMWYhMAXwgGustWvzLHMbMNVaO7NbE6bxEyOq\nEbmISIYjtqIxZgYwzlo7FbgKuDvPMicA07s/Xia/F9PphyIiWQppxTnAYgBr7RagxhjTP2uZ24Gv\ndXO2HJpaERHJVcjUyjBgfdrtusR9DQDGmCuBFcD2Ql6wpqaSYLBrpxDuIYbn91NbW92l5xebcnWO\ncnVeX82mXJ3T3bkKmiPPkvrkY2PMIODTwFnA0YU8+cCB5i68ZJw/cUFQXV1jl9dRLLW11crVCcrV\neX01m3J1TldzdVT+hcxT7CQ+Ak8aAexKfD0bqAVWAY8CExMHRrud5yXOI/f5jrywiMi/kEKKfClw\nKYAxZiKw01rbCGCt/b219gRr7RTgIuAla+0XixH0cJFrjlxEJN0RW9FauxpYb4xZTfyMlQXGmCuN\nMRcVPV2aaFRFLiKST0Fz5NbaG7Pu2phnme3AzPceKb9YTJfoi4jk40wrpopcI3IRkQzOtKKmVkRE\n8nOmFVMHO/3ORBYR6RHOtKLmyEVE8nOmFTVHLiKSnzOtGI2AH09TKyIiWZxpxVjUA9CIXEQkizOt\n6EVj8f+ryEVEMjjTirGIilxEJB9nWjFZ5KjIRUQyONOKXiQa/78OdoqIZHCmFTW1IiKSnzOtmDrY\nqRG5iEgGZ1pRI3IRkfycaUUvcR65DnaKiGRyphU1tSIikp8zrRgN6/RDEZF8nGnF5NSKRuQiIpnc\nacWYRuQiIvk404rJqRWNyEVEMjnTismDnfh8vRtERKSPca/INSIXEcngTCvqYKeISH7OtOLhqRVn\nIouI9AhnWlFTKyIi+TnTirqyU0QkP2daMfXBEipyEZEMzrSiplZERPJzpxV1ZaeISF7OtOLhEbku\nCBIRSedMkcciifcj19SKiEgGZ1pRc+QiIvk504oqchGR/NxpxZimVkRE8gkWspAxZiEwBfCAa6y1\na9MemwXcBkQBC3zGWhvr7qBeJBr/v4pcRCTDEVvRGDMDGGetnQpcBdydtcj9wKXW2jOAamBet6eE\n1OmHvkCgKKsXEXFVIcPbOcBiAGvtFqDGGNM/7fFJ1tq3E1/XAYO7N2Kc5shFRPIrZGplGLA+7XZd\n4r4GAGttA4AxZjhwNvD1jlZWU1NJMNj5UXVFWQkApRWl1NZWd/r5PUG5Oke5Oq+vZlOuzunuXAXN\nkWfJuSLHGDMEeAz4vLV2X0dPPnCguQsvCc1NLQCEIlHq6hq7tI5iqq2tVq5OUK7O66vZlKtzupqr\no/IvpMh3Eh+BJ40AdiVvJKZZngS+Zq1d2ul0BdL7kYuI5FdIKy4FLgUwxkwEdlpr03+d3A4stNY+\nVYR8KclPCCKgIhcRSXfEEbm1drUxZr0xZjUQAxYYY64EDgJ/Bj4JjDPGfCbxlIestfd3d9DkiNyn\ng50iIhkKmiO31t6YddfGtK/Lui9OB5LvfqgRuYhIBmdaUSNyEZH83GnFmD7qTUQkH2daMTUi19SK\niEgGd1oxeYm+PlhCRCSDc0WuS/RFRDI504pjRsff/XDocI3IRUTSOVPk498fAWDQUSpyEZF0zhS5\nplZERPJzpxVV5CIiebnTil78vVZ0HrmISCZnWtGXHJH7NEcuIpLOmSLX1IqISH7utKKKXEQkL3da\nUUUuIpKXO62oIhcRycudVozFr+zUWSsiIpmcacXDZ604E1lEpEe404qpTwgK9G4OEZE+xqEiT3z4\nsqZWREQyuNOKqamV3o0hItLXFPThy31BaNYc2LiO8KRTezuKiEif4kyRR0+cAIsX49U19nYUEZE+\nxZ2pFRERyUtFLiLiOBW5iIjjVOQiIo5TkYuIOE5FLiLiOBW5iIjjVOQiIo7zeYkPNRYRETdpRC4i\n4jgVuYiI41TkIiKOU5GLiDhORS4i4jgVuYiI41TkIiKOc+aDJYwxC4EpgAdcY61d28t5vg9MI74N\nbwMuACYB+xKL/MBa+3gPZ5oJPAy8mrjrb8D3gV8BAWAX8AlrbVsP57oK+ETaXZOBdUAVcChx35es\ntet7MNME4I/AQmvtj4wxI8mznYwxlwPXAjHgfmvtz3sh1yKgBAgDV1hrdxtjwsBzaU+dY62N9mCu\nX5Bnf+8D2+thoDbx8CBgDfBd4j8Lyf2rzlp7WZFzZffDWoq4fzlR5MaYGcA4a+1UY8zxwP8AU3sx\nzyxgQiLPYGAD8AzwVWvtn3orV8IKa+2lyRvGmEXAj621Dxtjvgv8B/DTngyU2Dl/nsgzA/gIcCLw\naWvtpp7MkshQBdwDLEu7+1tkbSdjzIPAN4BTgRCw1hjzqLV2fw/mupX4D/j/M8YsAK4DvgwctNbO\nLEaOAnNB1v6eWK5Xt1d6QRtj/gd44PBDPba98vXDMoq4f7kytTIHWAxgrd0C1Bhj+vdinpVAcoep\nJz6yDPRenA7NBJYkvn4MOKv3ogDxHffbvZyhDZgP7Ey7bya52+k0YK219qC1toX4CPiMHs71eeCR\nxNd1wOAivn578uXKpy9sLwCMMQYYaK19sYiv3558/TCTIu5fTozIgWEc/rMI4jv0MKChN8Ik/oRN\nTglcBTwBRIGrjTHXAXuAq621e3sh3gnGmCXE/6y8BahKm0rZAwzvhUwAGGNOAXYkpgYAvmWMOQrY\nAlyb2JmLzlobASKJDEn5ttMw4vsaWff3WC5r7SEAY0wAWED8LweAcmPMQ8Bo4BFr7R09mSshY3+n\nD2yvNNcQH60nDTPG/B4YQXxk/L9FzJWvH84p5v7lyog8m6+3AwAYYy4k/g91NfH5rxuttbOBl4Fv\n9kKk14mX94XAp4hPZ6T/su7t7fYZ4BeJr+8CbrDWTic+P7igt0Ll0d526pXtlyjxXwHPWGuT0wjX\nA/8JnA1cboyZ3MOxCtnfe2t7lQJnWmufTdy1D/g68O/Ej2V92xhT9AFNVj+k6/b9y5UR+U7iv72S\nRhA/YNBrjDHnAF8D5llrD5I5f7iEHp6HBrDWvgP8LnFzqzFmN3CKMaYiMdo9miP/eVxMM4EvAFhr\nH027/zHgo70RKE1Tnu2Uvd8dTfzgWU9bBLxurb0leYe19t7k18aYZcAHiB9A7hFpv1Dg8P7+e/rG\n9poBpKZUrLWNxLchwF5jzDrgOIrYIdn9YIwp6v7lyoh8KXApgDFmIrAz8Y/TK4wxA4AfAP+WPDBh\njHnEGDM2schMoDcO4l1ujLk+8fUwYCjxHfiSxCKXAE/1dK5EnhFAk7U2ZIzxGWOeNsYMTDw8k17Y\nXlmeJnc7vUD8F+FAY0w/4vOXq3oyVOKshpC19ua0+4wx5qHEdgwmcr3a7kqKkyvf/t7r2yvhFGBj\n8oYxZpYx5o7E11XAycBrxXrxfP1AkfcvZ97G1hjzf4HUn+HW2o1HeEoxs/wn8T8l03eGRcT/hGoG\nmoifkbGnh3NVAw8BA4FS4tMsG4AHgXLgzUSucE/mSmSbBNxqrT03cfsjwFeIzyW+A1xlrW3uwSy3\nA8cSP6XvHeBy4tM+GdvJGHMpcAPx017vKebcaju5hgCtHD4etNla+3ljzPeA2cR/HpZYa7/Tw7nu\nAW4ka3/vA9vrYuL7/V+ttb9LLBckfvaKIX5Swk+ttYvyrbObcuXrh08lMhRl/3KmyEVEJD9XplZE\nRKQdKnIREcepyEVEHKciFxFxnIpcRMRxKnIREcepyEVEHPf/Afdzy0KORyQ5AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tkToA3CjBxJe",
        "colab_type": "code",
        "outputId": "b0930a09-31e6-4eeb-b099-035d807097dc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "source": [
        "plt.plot(history['loss_train'],'b')\n",
        "plt.plot(history['loss_val'],'r')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f1844035358>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XuYHHWd7/F3VXXPNTOTIekkJFxC\nAvwgEgh3QZAgLAgnyLpBPSsiu7DHPSy6uuDuknXFXWGRg+sioivLo6B4QRTkalYQoiYQIxeBQAi/\nECAk5EKaZHKZe1/q/FHVM93TPTPdk+mZqfXzeh4epqtqqr9dU/PJb751c3zfR0REossd7wJERGTf\nKMhFRCJOQS4iEnEKchGRiFOQi4hEXGys3zCZ3Dvi02RaWxtoa+sczXJGzUStTXVVZqLWBRO3NtVV\nmZHWlUg0OYPNi9SIPBbzxruEQU3U2lRXZSZqXTBxa1NdlalGXZEKchERKaYgFxGJOAW5iEjEKchF\nRCJOQS4iEnEKchGRiFOQi4hEXGSC3H17EyxZAh0d412KiMiEEpkgr334QbjxRmpWPTXepYiITCjD\nXqJvjGkAvgdMB+qA66y1j+TN3wBsAjLhpIuttZtHu1ByD8BIZ4ZeTkTkj0w591q5AHjWWnuTMeZg\n4FfAIwOWOc9a2z7q1eVzw9sMZLNVfRsRkagZNsittffkvTwQeLt65QzBDbtACnIRkQJl3/3QGLMS\nOABYVGL2bcaY2cCTwBJr7aB3OGxtbRjZTWOaGwBoaaqFRFPl3z8GEqqrIqqrchO1NtVVmdGuq+wg\nt9aeaoxZAPzQGHNMXlhfC/wS2Ak8ACwG7h1sPSO9rWRdRy9NwO5dHfQm945oHdWUSDSRVF1lU12V\nm6i1qa7KjLSuocJ/2LNWjDHHG2MOBLDWvkAQ/oncfGvtXdba7dbaNLAUmF9xheUIWyuOWisiIgXK\nOf3w/cDVAMaY6cAk4N3wdYsx5lFjTE247BnAy9UoVD1yEZHSygny24BpxpgVwC+AK4FPGmM+bK3d\nTTAKX2WMeQpIMkRbZd8qVZCLiJRSzlkrXcDHh5h/C3DLaBZVkoJcRKSkyFzZ6SvIRURKikyQ62Cn\niEhpkQtyjchFRAopyEVEIi4yQd6TCkr1MwpyEZF8kQny3/0+DsCbr49zISIiE0xkgry7Nyi1p1sj\nchGRfJEJcscLS80Mej8uEZE/SpEJ8tzBTvXIRUQKRSbI+0bkOmtFRKRAZIJcpx+KiJQWuSBXa0VE\npFBkgtyNaUQuIlJKZIJcrRURkdIiE+SO5wDgK8hFRApEJsj7RuTqkYuIFIhMkPeffqgLgkRE8kUw\nyDUiFxHJN+yj3owxDcD3gOlAHXCdtfaRvPlnAzcAGWCptfa6ahSaC3L1yEVECpUzIr8AeNZaewbw\nUeA/Bsz/BrAYeB9wjjFm3uiWGNCIXESktHIevnxP3ssDgbdzL4wxc4Cd1tpN4eulwFnAK6NcZ/+j\n3jKZUV+1iEiUDRvkOcaYlcABwKK8yTOAZN7r7cDc0SmtkBsLTj/UiFxEpFDZQW6tPdUYswD4oTHm\nGGttqdNHnOHW09raQCzmVVIjAJNaGgGIx1wSiaaKv38sqK7KqK7KTdTaVFdlRruucg52Hg9st9Zu\nsta+YIyJAQmC0fcWglF5zqxw2qDa2jpHVGhndw8Aqe5eksm9I1pHNSUSTaqrAqqrchO1NtVVmZHW\nNVT4l3Ow8/3A1QDGmOnAJOBdAGvtBqDZGDM7DPhFwGMVV1gGnUcuIlJaOUF+GzDNGLMC+AVwJfBJ\nY8yHw/lXAHcDK4B7rLXrqlNpWKqvHrmISL5yzlrpAj4+xPzlwCmjWVQpuvuhiEhpurJTRCTiIhfk\njoJcRKRAZIJcrRURkdIiE+Q62CkiUlpkgjz3YAmNyEVECkUmyHOtFUfnkYuIFIhMkPedtaLWiohI\ngcgEef+IXEEuIpIvMkGuEbmISGkRCvLgYKdG5CIihSIT5DqPXESktOgFuVorIiIFIhPkfa0VBbmI\nSIEIBbnOWhERKSUyQd53+qFG5CIiBSIT5E74nE+NyEVECkUmyHWwU0SktMgEuRdXa0VEpJTIBHnu\nNrYKchGRQsM+sxPAGHMTcHq4/FestT/Pm7cB2ARkwkkXW2s3j26ZuteKiMhghg1yY8yZwFHW2lOM\nMVOA54GfD1jsPGttezUKzPHCSjUiFxEpVE5rZTnwkfDrXUCjMcarXkmluZ5DFkdBLiIygOP75T+o\nwRjzKeB0a+0ledM2AE8Cs8P/L7HWDrrSdDrjx2KV/zuwezc0To6xrvW9zNv5ZMXfLyIScc5gM8rq\nkQMYYy4ELgfOGTDrWuCXwE7gAWAxcO9g62lr6yz3LQu0t0M9Ln42QzK5d0TrqKZEokl1VUB1VW6i\n1qa6KjPSuhKJpkHnlXuw81zgC8AHrbW78+dZa+/KW24pMJ8hgnykXBeyuLhqrYiIFBi2R26MaQG+\nCiyy1u4cOM8Y86gxpiacdAbw8uiX2R/kuiBIRKRQOSPyjwFTgZ8aY3LTlgEvWWvvD0fhq4wxXQRn\ntIz6aBzA84Ig18FOEZFCwwa5tfZ24PYh5t8C3DKaRZWi1oqISGmRubIzF+QakYuIFIpMkDtOGOQo\nyEVE8kUmyCE3Ii//vHcRkT8GEQxyjchFRPJFLsh1sFNEpFCkgtx31CMXERkoUkGu1oqISLHIBbnr\nZ4ZfUETkj0i0gtzx1CMXERkgWkGu88hFRIpEKsh9R2etiIgMFKkg14hcRKRYpIJcI3IRkWLRCnKN\nyEVEikQqyLOOi6sgFxEpEKkg9x1dECQiMlC0ghyNyEVEBopUkKu1IiJSLFJBrtaKiEixch6+jDHm\nJuD0cPmvWGt/njfvbOAGIAMstdZeV41CITz9UCNyEZECw47IjTFnAkdZa08BPgh8fcAi3wAWA+8D\nzjHGzBv1KkPqkYuIFCuntbIc+Ej49S6g0RjjARhj5gA7rbWbrLVZYClwVlUqBXxXQS4iMtCwrRVr\nbQboCF9eTtA+yd1LdgaQzFt8OzB3qPW1tjYQi3kjKBXeDkfkiUTTiL6/2lRXZVRX5SZqbaqrMqNd\nV1k9cgBjzIUEQX7OEIs5w62nra2z3Lcs4jsuHlmSyb0jXke1JBJNqqsCqqtyE7U21VWZkdY1VPiX\nddaKMeZc4AvAedba3XmzthCMynNmhdOqwnfCcn2/Wm8hIhI55RzsbAG+Ciyy1u7Mn2et3QA0G2Nm\nG2NiwCLgsWoUCnlBnlWfXEQkp5zWyseAqcBPjTG5acuAl6y19wNXAHeH0++x1q4b9SpDBUHujazP\nLiLyP005BztvB24fYv5y4JTRLGowvqsRuYjIQJG7shPAT+sBzCIiOREL8qCd4mc0IhcRyYlYkAfl\nZlIKchGRnGgFedgjz6YV5CIiOZEKchwFuYjIQJEK8r6DnRldECQikhOtIHdzQa4RuYhITqSCHB3s\nFBEpEqkg7x+Rq7UiIpITqSDXwU4RkWKRCnL1yEVEikUqyDUiFxEpFqkg14hcRKRYpIIcXdkpIlIk\nWkGu1oqISJFIBblaKyIixSIV5Og8chGRItEKckcjchGRgaIV5LmDnRqRi4j0KefhyxhjjgIeBG62\n1n5zwLwNwCYg9/y1i621m0exxn651ooOdoqI9Bk2yI0xjcCtwBNDLHaetbZ91KoajE4/FBEpUk5r\npQc4H9hS5VqGlTtrhYweviwikjPsiNxamwbSxpihFrvNGDMbeBJYYq0dtInd2tpALOZVWmfADb5v\nUkMtiUTTyNZRRROxJlBdlZqodcHErU11VWa06yqrRz6Ma4FfAjuBB4DFwL2DLdzW1jnydwpH5Ht2\ndZBM7h35eqogkWiacDWB6qrURK0LJm5tqqsyI61rqPDf5yC31t6V+9oYsxSYzxBBvk90QZCISJF9\nOv3QGNNijHnUGFMTTjoDeHnfyxpE38FOnX4oIpJTzlkrxwNfA2YDKWPMRcBDwJvW2vvDUfgqY0wX\n8DzVGo1DX5CT1YhcRCSnnIOdzwELh5h/C3DLKNY0OLVWRESKROrKTsfTeeQiIgNFKsj7RuRqrYiI\n9IlkkKPWiohIn2gFuaceuYjIQJEKckf3IxcRKRKpINfphyIixSIV5DprRUSkWKSCXCNyEZFikQry\n3IicrHrkIiI5kQry/ke9aUQuIpITrSD3dB65iMhAkQryXGtFV3aKiPSLVpDr4csiIkUiFeS5Hrnj\nK8hFRHIiFeRO7lmf6pGLiPSJVpDrXisiIkUiFeR6sISISLFIBbkT05WdIiIDDfuoNwBjzFHAg8DN\n1tpvDph3NnADkAGWWmuvG/UqQ44u0RcRKTLsiNwY0wjcCjwxyCLfABYD7wPOMcbMG73yCvVfoq8g\nFxHJKae10gOcD2wZOMMYMwfYaa3dZK3NAkuBs0a3xH66IEhEpNiwrRVrbRpIG2NKzZ4BJPNebwfm\nDrW+1tYGYrnTCCv0VhjkNTGPRKJpROuopolYE6iuSk3UumDi1qa6KjPadZXVI6+AM9wCbW2dI195\nGOS9Xb0kk3tHvJ5qSCSaJlxNoLoqNVHrgolbm+qqzEjrGir89/WslS0Eo/KcWZRowYwW9chFRIrt\nU5BbazcAzcaY2caYGLAIeGw0CitFPXIRkWLDtlaMMccDXwNmAyljzEXAQ8Cb1tr7gSuAu8PF77HW\nrqtSrRqRi4iUUM7BzueAhUPMXw6cMoo1DcrVBUEiIkWidWWnHiwhIlIkUkHe94Qg3cZWRKRPpILc\nDYPcUWtFRKRPpIJct7EVESkWySDH98e3EBGRCSRSQZ47a0WtFRGRfpEKcp1HLiJSLFJB3j8iz4xz\nJSIiE0ekgjz38GVdoi8i0i9SQa4euYhIsUgGuXrkIiL9IhXkuYOdjq7sFBHpE60g7xuR6zxyEZGc\nSAW5F9O9VkREBopUkDu614qISJFIBbkOdoqIFItkkOtgp4hIv0gGuXrkIiL9IhXk6pGLiBQb9pmd\nAMaYm4H3Aj7wWWvtM3nzNgCbgNwNUC621m4e3TIDXlwjchGRgYYNcmPMGcBh1tpTjDFHAndQ/LDl\n86y17dUoMJ8uCBIRKVZOa+Us4AEAa+1aoNUY01zVqgahe62IiBQrp7UyA3gu73UynLYnb9ptxpjZ\nwJPAEmvtoJdetrY2EAvvYlipzj3BoN9zIJFoGtE6qmki1gSqq1ITtS6YuLWprsqMdl1l9cgHcAa8\nvhb4JbCTYOS+GLh3sG9ua+scwVsGmsIReTaTIZncO+L1VEMi0TThagLVVamJWhdM3NpUV2VGWtdQ\n4V9OkG8hGIHnzAS25l5Ya+/KfW2MWQrMZ4gg3xdqrYiIFCunR/4YcBGAMeY4YIu1dm/4usUY86gx\npiZc9gzg5apUSv9ZKzrYKSLSb9gRubV2pTHmOWPMSiALXGmM+Qtgt7X2/nAUvsoY0wU8T5VG46Ar\nO0VESimrR26tvWbApBfz5t0C3DKaRQ1Gpx+KiBSL1JWduLkg18OXRURyohXkXnDaog52ioj0i1aQ\n50bkKMhFRHKiGeTqkYuI9FGQi4hEnIJcRCTiohXkTnB3AB3sFBHpF7kgz+LQ2+2TSo13MSIiE0O0\nghzwHRf8LOvWRa50EZGqiF4aui4uWV56KXqli4hUQ/TSsC/IR3ZPcxGR/2kiF+SO5+KRZfXqyJUu\nIlIVkUtDx3Wpr8vy8sseOnlFRCSCQe67Lg11GTo6HN58c+DDikRE/vhELshxXRrqgqH473+vPrmI\nSOSC3G9oINH2GpfxXR76Sf/J5FdfXcvixfXs3j2OxYmIjIPIBXn7DV/Ficf4Ln/FY6ta8T7+Fzz3\ntM8PflDDihUxLr20nu7u8a5SRGTslPWEoImkd9GHaDtmARuvvo2a3yzjPY//nG1vHA18iQULMqxc\nGeP00xu56qoeTj45w5o1Ht3dcNFF6dwV/kXuuy/GV75Syz0/6WDuXJ+CBf0Br6vE2b0Lv2Vy1d9n\nXGQy1N5/L6TT9Pzvi/smO+178Rsa++6hUyCbxbOv4q1fh9/cQnreUfhTpuBu2kjsNUvmoNlk5syF\nWP8u7L6zjfivn8Db8CZks/hNzWQPOojUSe/F3baVmt/+mviyxyFeQ3rePDo/cxX+lCk4He34zS2j\n93l9H6d9L+72d3B27sRJpcjMmUt2xv6j9x5Rkc0Se3k12aZmsofMGd9aenogHi+9v+Vxkklidi2p\nY4+HxsYxKm7fRC7IAbIHHkTrnTdw+nt6Wdm5gEve+DLN07ew8OJj+EWzi7viKTo+W89pfIs0cWro\nof7HP+KCr56Mf+ihBevq7YXrr6+lbvMbzPqTRTQvNOz5zvfB86j92U+Y9MVr6Pzbq+m69DJqfvck\n3tq1ZA49jN7z/hd0duJ0dOAnEiP+LE4yCZ/5PFOX3U/ygk/gfP1G/KZm6OnBe3093sa3SM97D9np\nM4itXUPsxRcgnaLngg/jT5vW/yE8r//BG7t34W7bRuZw0/9GqVQQnDW1Q++cHR3UPv4o8d8sA6Dz\n89eQnXXA8B/E93E3vkXNE78CP0t6wXFkZ+xP/OlV1N/6deIvrw5Wv2UzmZmz4N67mbp8Oem5h9J7\n7vk4nR1kE9PIHDIHfJ/6275F/KUXC9/CdQvus5OZOYuuyz6FP3kyNY/9NzW/ehTH94cuM1xHzYrf\nUPfjH0LMw21rI/2e+aROfi+YQ2l8fQPunj2QTkM6RXbadNILjiNzuCF96OHQ0ID7zjacXbvIHHZ4\nfzC0tzPpn/+Rugfuw+nsLHrvzIEHkZ0yhYw5ktSJJ0MmEyzn+6TedxrpBccFg4Z0Gqd9L/Sm8DZu\nwFv/Gt5bG6DGpSHjkJ2aIJuYht/c3Lft/cmTSc8/pn/QEf48vDffwG9sxG+ZjF9bi7v9Hbwtm3G3\nbMHdshmnqwu/vg7q6nH27Mbd/Dbpo+aTPuY4nJ5uvPWvQaqX3jP/BACns4P0scfht+4XvE9nJ3zh\nRlp/+jMysw8hO+tA/LpanM5OvLfeIvbSC7htbfiuS/fFn6T37HPJTp0KflAjvo+T6iW25iWc9nZ6\nzzgTf1IT1NaQmXNo0TaslLN3D/W3f5vahx/EW7sGPA9/0iRIpcnuvz+ZQ+bgNzbi7tmDs6sN0pmg\nlkwGv66Ongv/jI6/X0L2wIOKB3S+T83DD1D/w+8Tf3oVvaefQfdH/xxwSB+zgOxBB/dtI2/rZtxk\nkvRR8yHRtM+fq+hz+sPs+ADGmJuB9xJs/s9aa5/Jm3c2cAOQAZZaa68bal3J5N7h33AQiUQTyeTe\nvtcrV3qs+toz/MPKxbRmdhQt//CcT7PppD/lA/dfzRE9L9LutbD2H/+LN6cez9FHdDEjtZkXvv8K\ny3++i8u4gwPYDEDH5z5P+rgTaL78Epzwpi5+LIaTTvetu/Ov/4bapY/gbn+Hjn/+FybNPoD2DW+T\nOuEk/Lp68Dwy5gjo7sbd8S7ZWQdQd9ed1D78IF2Xf4reD56Pt87S8pEL8bZtZTfNtLAH3/PITpse\njOYy/Y+08z2v8HUsRvqIeVBbE4S775OdmsDfbwre+nU4qRQ9Hzyf2ov/nK5lv6X2vp/h7gkOIKQP\nO5yeY05grX8Es4+fTH1yI/FVv8Pp7CT22jqczo6+98k2NePvtx9ORwep955KtqUFXJfUqafhvfE6\nNb/9Ne62rbjbtuL09g76s+te/FHiq1bibX67b1pqwbHE1r6C09NT8nt6Fl1I6sSTcdp2ErOv4r6b\nJJuYRvqo+Xgb3qT2kQcLAjN1/In0fOjDpOcfjR+L4+7ZhffqWuLPPk12/5mkTjyZ3rPPwa+to+7u\nH9J407/h1zeQOXg28WefHrL+vu1eW0v6yHnEVr+Ik82SbW0lc9Bs/IYGvA1v4m3dQubg2aQPN0HY\n7jcFPxYj9vJqYi+txt29C2eQ3l9mxv6kjzuB+O+exG1rG7aWou8/eDbZ5ha8rVtwOtpxuroqXke5\nsq2t+I2TcHfuwOnsxK+rK/m5MgcdTOrU04g9/xwx+2pF79F7+hmk3nc62f2m0HvWn5A94ECc3buI\nvbIGHIfMjP2D0Pd9YuvXUf9f/4m3fh3EYrjvvovb3YXf24vT04NfXx/8Qxf+tYTr4W7a2Pc7AeDX\n1IDjkD5yHqkTT6Zm2ePEXl/fPz8Wg3gc34vhT5lCT7yRhvVrgs85cxbels2Fn33adJyuLty9e/qm\ndV7xGRr+8xsFOVauRKJp0NbAsEFujDkD+Htr7SJjzJHAHdbaU/LmvwKcC2wGfgv8tbX2lcHWN5pB\n3iedJvbsM3gb3sDJZEgfOY+mz11J7NW1fYs8NfUCTnj3UWoZ/Jf1Wv6Vy7iD2bwFQMaNcd1h3+MT\n/IiZmY3YQ8/jx6uP4ap3lrB/5m18xyE7qRlvb+kjrD2TE8Q79+D29pCdPBl3167+kmcdhLNnD97e\nXSzhBm7m77iar3HVkb+gZddGemccSO/hRxI75ADif3gGd+fOcKR0LHR1UnfvPcTWvgK9vaSPPgbi\nNUH4J5P0HDiHHq+ByWtW9b1fZtp0uhecSGbnXiateRa3q6OgVt918WvryM6YQc+HL2LvmefjvLCa\nqf9+LU4shl9bW7SjAmRdj9TUGTizZuAfcCDdpy3Er6ujZs1q3OQ7pKfPYusFf0nd/Lk0bn6Nxi8u\nIX3MsTRe+dckJ03F2bED77V1+C0tuFu34G3aiNPZSeq004NfvCE4O3ZQ8/ijwS+fOZLOIxbQ1RX8\nYTJpUhkdsdy+7zjQ1UXMrqW1o422xlayk1uDtk0shvvWW8RXP4/3xuvEVz5FbO0aUsefQOaQucSe\newZv21acri6yjZPouvQyOpZcix+vKXob3ydoNax5idial/Dr6vDrG3E6O6j79a+oe/yXuLvaSCVm\n0HP0ccTq42QPPIjM3EPJzD6EyftPYdfmJO6Od3GT23H27Alqdxy819dT+9+PgO+TmTkLf1ITmdmH\nkDnc4HR34+zejdPdRXbadDIzZ5LdfxbZmTPxGxpxerqhqxsa6snMmEn86VV4G97Er6sle/BsyGSp\n+c0T+PUN+DVx4n94LhjNd3biN7cQu/ACkp/6W5yebtxkEqe7C7+hMdiXapt58UUXUilOanuMujcs\n7q62vrp9xwHXJXO4wa+ppWb5r4PPs349NSt+U/jjcpxh/+JKNbfiOT7Z/fYj1tJMKp2l94IL6fw/\nVwQ7xYCfv7N3D05nJ9lJTfiNk9i9G5JJl/Z2aG3JcPDKn9Ly8E+CbRj+heb3pulZv5mmnh2snP4h\nDvrRl6idfyjx3z1F7IXnwfeJP7Wc2Gvr8BsnkU0kyMw6gOzMWXR/7ONMOWH+uAT5l4GN1trvhK9f\nBU6y1u4xxswB7rLWnhbOWwK0W2tvHWx9VQnyErw31tP0fy8nc+jhdH/iUnpPOY1Vt75Aw313s1/m\nXda83siWzDRe5BhO/1iCj/5dggdeNqz45isseuFGkkzlXi5iGWcVrtfzmZNdz/X+F/g+l/I8x/JZ\nbmE709jONE7m9/g4TGYXH2AZ25nGWxzMSTzNc5zAzd5VXJH9Ngv9ZTTSwd/yDe5wLuff/72Hq6+u\nK/ocsZhPPF76Mzp+ljgpep3avmnZLPT0ODhkuYh7mRVP8qpzBCuc99PRE6zII81cXufUKa+S2rGX\nNlpZzvtppwnX9YnFoLc3t8/4OA7U1vjMZgNxUjSxl/enl7EpM5MH+FM6CH5B4nGfVKp/X6up8fPW\nE7zuD1eH4A+8wVVyaCKVgkym/xtiMb+vfZ6/iw/3te/31zXYsrV+N93U5727j4OPv4/nDnikmc0G\n3uQQsnh4np/rloUcHKdwm+VvI89Pk3W8MTmmU2jwn2UqBel0UI/n+dTUlFyspMOzrzLLf5u5/nrO\nzzxCE3top4mX3aPIEGN/fyuNfju+47AzO5nv+5/kKU4jFsu9z/D7WL7e3v5a88XjhT+HTCb4XFPr\nO3i3axLxuJ9/qGZQjgNf/GIP11xTNy5BfjvwC2vtg+HrFcDl1tp1xphTCUbrHw7nXQ7Mtdb+02Dr\nS6czfiw2/ud/v/46rFgBZ54JBx9cOG/zZrj7bmhogHPPhTvugI0bYc4cuOQS2L4drr8+aKE2NQUt\n53Qaamrg6KODtmEyCXV1wbo2bQoCNmwJks1Cc5PPlJY023bEOeccuOYa+NKXYNmyYOAwaVKws7zz\nTrDjVGLWLJg5E6yFPf1/1dHSAjNmBOs89tjgM9x0Ezz2GNTWBu/T2xu8b0tL8Pm7uoLPU6obcNhh\nMH8+rF4N27YFy9bVBTtsKhX8V18PU6ZAWxt0dBSvYzBldPwKeF7wc2hsDN53504KrvzNz7bhvq5k\n2dFet+MEP/t0GnbsKPzZD9wmw72eCOJxOC5s/f/hD8H+VQ2xGOQOf61fz4hucx2LwbRpMH168Hu9\nc2fwu75jR/G2PfFEuOEGuPFG+NWvyvu303Xhn/4JPvShymsLjWqQPwlcNkiQ/xUwZ6ggH6sR+Vib\nqLWprspM1Lpg4tamuioz0rqGGpGX87fgFmBG3uuZwNZB5s0Kp4mIyBgpJ8gfAy4CMMYcB2yx1u4F\nsNZuAJqNMbONMTFgUbi8iIiMkWFb9NbalcaY54wxK4EscKUx5i+A3dba+4ErgLvDxe+x1q6rWrUi\nIlKkrAuCrLXXDJj0Yt685cApiIjIuIjcvVZERKSQglxEJOIU5CIiEacgFxGJuLJumiUiIhOXRuQi\nIhGnIBcRiTgFuYhIxCnIRUQiTkEuIhJxCnIRkYhTkIuIRFxZN82aCIZ6APQ41XMTcDrBNvwK8CHg\neCD3FOivWmt/McY1LQR+BqwJJ70E3AT8APAI7iN/ibW29NOOq1fX5cAleZNOAJ4FGoHcc4OuttY+\nN4Y1HQU8CNxsrf2mMeZASmwnY8zFwOcI7vx5u7X2u+NQ151AHEgBn7DWbjPGpICn8r71LGtthc+S\n2qe6vkeJ/X0CbK+fAYlw9n7AKoKHw78E5PavpLX2I1Wua2A+PEMV969IBHn4AOjDrLWn5B4AzTje\ncdEYcyZwVFjPFOB5YBmwxFr7yHjVFfqttfai3AtjzJ3At6y1PzPG3ABcBnx7LAsKd87vhvWcAXwU\neA/wl9bal8eylrCGRuBW4Im+l9HIAAAECklEQVS8yV9mwHYyxtwFXAucBPQCzxhj7rfW7hzDuq4n\n+AX/qTHmSuAq4B8IbiO9sBp1lFkXDNjfw+XGdXvlB7Qx5g7gO/2zxmx7lcqHJ6ji/hWV1spZwAMA\n1tq1QKsxpnkc61kO5HaYXQQjy/F/EGlpC4GHwq8fBs4ev1KAYMe9bpxr6AHOp/BpVgsp3k4nA89Y\na3dba7sIRsDvG+O6/ga4L/w6CUyp4vsPplRdpUyE7QWAMcYAk621T1fx/QdTKh8WUsX9KxIjcoLH\nyeX/2Z0Mp+0pvXh1hX/C5loClwNLgQzwaWPMVcB24NPW2nfHobx5xpiHCP6s/FegMa+Vsh3Yfxxq\nAsAYcyKwKWwNAHzZGDMVWAt8LtyZq85amwbSYQ05pbbTDIJ9jQHTx6wua20HgDHGA64k+MsBoM4Y\n82PgYOA+a+1/jGVdoYL9nQmwvfJ8lmC0njPDGHMvwaMqv2Wt/VEV6yqVD+dWc/+Kyoh8oDKeWV19\nxpgLCX5Qnybof11jrf0A8ALwL+NQ0msE4X0hcClBOyP/H+vx3m5/BXwv/PoWggd3v5/wyVPjVVQJ\ng22ncdl+YYj/AFhmrc21ET4PfAo4B7jYGHPCGJdVzv4+XturBjjNWvvrcNIO4IvAnxMcy7rOGFP1\nAc2AfMg36vtXVEbkQz0AelwYY84FvgB80Fq7m8L+4UOMcR8awFq7GbgnfPm6MWYbcKIxpj4c7Y73\nw7EXAp8BCB8TmPMw8LHxKChPe4ntVOrh4qvGobY7gdestf+am2CtvS33tTHmCWA+wQHkMZH3Dwr0\n7+/3MjG21xlAX0slfMbwneHLd40xzwJHUMUMGZgPxpiq7l9RGZEP+gDo8WCMaQG+CizKHZgwxtxn\njJkTLrIQGI+DeBcbYz4ffj0DmE6wAy8OF1kM/HKs6wrrmQm0W2t7jTGOMeZxY8zkcPZCxmF7DfA4\nxdvp9wT/EE42xkwi6F+uGMuiwrMaeq21X8qbZowxPw63Yyysa82gK6lOXaX293HfXqETyXscpTHm\nTGPMf4RfNwILgKo9W7hUPlDl/Ssyt7E1xtwI9P0Zbq19cZhvqWYtnyL4UzJ/Z7iT4E+oTqCd4IyM\n7WNcVxPwY2AyUEPQZnkeuAuoA94K60qNZV1hbccD11trzwtffxT4R4Je4mbgcmtt5xjW8jVgNsEp\nfZuBiwnaPgXbyRhzEfD3BKe93lrN3uogdU0Duuk/HvSKtfZvjDH/D/gAwe/DQ9bafxvjum4FrmHA\n/j4BttefEez3T1pr7wmXixGcvWIITkr4trX2zlLrHKW6SuXDpWENVdm/IhPkIiJSWlRaKyIiMggF\nuYhIxCnIRUQiTkEuIhJxCnIRkYhTkIuIRJyCXEQk4v4/t6XIQVhYwB8AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NJPdR4IFb-uc",
        "colab_type": "code",
        "outputId": "4409d520-8b2b-42fa-a9ac-61875d25abdf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        }
      },
      "source": [
        "plt.plot(history['learning_rate'])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f1843f6c550>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xt8XHWd//HXZCb3S5OmadOU0NLS\nfuiFiwUWCmqrrbAqq7sL617wgovrPpDdRf25yu/h46crKLrrruyyN9fHT3BFUVQeuPCzIgoqSLm0\nFYq25VNaWnpJ26RtmqRt7jO/P+akTEMmnUkzOZPM+/l48EjmnPM98z7DdD453+8534kkEglERKSw\nFYUdQEREwqdiICIiKgYiIqJiICIiqBiIiAgQCzvAWLS1dY35Eqi6ugra20+MZ5xxk6/ZlCs7ypW9\nfM021XI1NFRH0q0ruDODWCwadoS08jWbcmVHubKXr9kKKVfBFQMREXk9FQMREVExEBERFQMREUHF\nQEREUDEQERFUDEREhAIrBntbj/HtR14irmm7RUROUVDF4JktB/nuT52dLZ1hRxERySsFVQxm1pUD\n0HLoeMhJRETyS0EVg9n1FQC0HFYxEBFJVWDFoBKA/Yfzb+IpEZEwZTRrqZndCVwOJIBb3H19yro1\nwB3AILDW3W9P18bMVgBfBvqBXuB97t5mZtcDHwXiwNfc/evjdYCpqsqLqasuVTeRiMgwpz0zMLOV\nwEJ3XwHcCNw1bJO7gGuBK4GrzGzJKG0+Drzf3d8CPA38hZlVAp8B1gCrgI+Z2fQzPrI0mmdVc7ij\nh96+wVw9hYjIpJNJN9Fq4IcA7r4VqDOzGgAzmw8ccfc97h4H1gbbj9jG3f/I3V8xswgwB9gLXAas\nd/cOd+8GniJZWHKieVY1CeDAEXUViYgMyaSbqBHYmPK4LVjWGfxsS1nXCiwAZqRrY2a/S/JMYSvw\nLeBPRtjH7NEC1dVVjHk+7+aZVQB09Q3S0FA9pn3kUj5mAuXKlnJlL1+zFUqusXzTWdpvyhll3cnl\n7v6ImRnwJeBWYFcW+wc4o28eam5MvoC+8zDLzq4d835yoaGhmra2rrBjvI5yZUe5spev2aZartEK\nSCbdRC0k/6of0gTsT7NuTrBsxDZm9gcA7p4AHgDeOMo+cqJ5VvLF0CCyiMhrMikGjwLXAZjZcqDF\n3bsA3H0XUGNm88wsBlwTbJ+uzd+Z2UXBfi8DHHgWuNTMas2siuR4wZPjdHyvU1tVSmVZjBZdXioi\nctJpu4ncfZ2ZbTSzdSQv/bzZzG4AOtz9QeAm4DvB5ve7+zZg2/A2wfobgf8wswGgm+Slpd1mdivw\nE5KXoX7O3TvG8RhPEYlEmD2jklf2ddI/EKc4VlC3WoiIjCijMQN3v3XYok0p654AVmTQBnffAFwx\nwvIfAD/IJMt4aKqvYPveDg62n+CshqqJeloRkbxVkH8WNwV3ImvcQEQkqTCLwQxNSyEikqogi8Fs\nnRmIiJyiIIvB9JpSSkui7NfspSIiQIEWg0gkwuzpFRw4coLBeDzsOCIioSvIYgDJcYOBwQRtR3vC\njiIiErqCLQZzgkHkfW3HQk4iIhK+gi0GZwUT1u1pVTEQESncYhDcbLa3TYPIIiIFWwxqq0qoKi9m\nr84MREQKtxhEIhHOaqik9Wg3PX0DYccREQlVwRYDeG3cYJ+6ikSkwBV0MWgOxg326IoiESlwBV0M\nhs4MNG4gIoWuoItB04xKIhEVAxGRgi4GpcVRZtVVsKftOIlEIuw4IiKhKehiAMmuou7eAY509oYd\nRUQkNAVfDJobktNSaBBZRApZwRcDDSKLiKgYnLy8dK/ODESkgBV8MaifVkZZSVQT1olIQSv4YhCJ\nRDhrZhUHj3TTPzAYdhwRkVAUfDEAaJ5ZRTyRoOXQibCjiIiEQsWAZDEAePVgV8hJRETCEctkIzO7\nE7gcSAC3uPv6lHVrgDuAQWCtu9+ero2ZNQP3AMVAP/Bedz9gZv3AUylPudrdJ6zPZu6sagBePdAF\nF07Us4qI5I/TFgMzWwksdPcVZrYYuBtYkbLJXcDVwD7gl2b2ANCQps3nga+5+/fM7Gbg48AngQ53\nXzWOx5WVsxqqiBZF2HVAZwYiUpgy6SZaDfwQwN23AnVmVgNgZvOBI+6+x93jwNpg+3RtPgI8EOy3\nDagfx2MZs+JYEXMaKtnTeoyBwXjYcUREJlwm3USNwMaUx23Bss7gZ1vKulZgATBjpDbuvg3AzKLA\nzcBtwfoyM7sPmAs84O5fGS1QXV0FsVg0g+gja2ioft2y8+bVs/vgMXricE7j69dPlJGy5QPlyo5y\nZS9fsxVKrozGDIaJjGHdyeVBIbgXeNzdHwsWfwL4FsnxhSfM7Al335DuSdrbx37VT0NDNW1tr+8O\nmlVbBsDzWw9QVRzOuHq6bGFTruwoV/byNdtUyzVaAcmkGLSQPAMY0gTsT7NuTrCsb5Q29wAvu/vn\nhla6+1eHfjezx4DzgbTFIBdSB5HfdMFEPrOISPgy+RP4UeA6ADNbDrS4exeAu+8CasxsnpnFgGuC\n7UdsY2bXA33u/tmhnVvSfWYWCfZxJbB53I4wQ80zK4kWRZJXFImIFJjTnhm4+zoz22hm64A4cLOZ\n3UDyCqAHgZuA7wSb3x+MC2wb3iZYfzPJ8YFfBI+3uPtHzGwP8Fyw7UPu/tw4HV/GimNRmmYkB5EH\n43GiRboFQ0QKR0ZjBu5+67BFm1LWPcGpl5qma4O7X5Fm/5/KJEeuzW2sZk/rMfYfOnFyNlMRkUKg\nP39TzAuuItL9BiJSaFQMUsxtTLkTWUSkgKgYpGhuqKIoEtEcRSJScFQMUpQUR2maUcHu1i7i8UTY\ncUREJoyKwTBzG6vp64+z//DxsKOIiEwYFYNh5jXWALBzv7qKRKRwqBgMM78pWQxe2d8ZchIRkYmj\nYjBM88wqimNFvLKvI+woIiITRsVgmFi0KHnzWdsxevv0ncgiUhhUDEawoKmGRAJ2qqtIRAqEisEI\nFjRNA2BHi7qKRKQwqBiMYMGcZDF4pUVnBiJSGFQMRlBXXUpddSk7WjpJJHTzmYhMfSoGaSxoqqHz\neB+HO3rCjiIiknMqBmnMPzluoK4iEZn6VAzSWDAnefPZDt1vICIFQMUgjbmzqokWRXRmICIFQcUg\njZLiKGfPqmL3wS76B3TzmYhMbSoGo5jfNI3BeIJXDx4LO4qISE6pGIxiQTBp3fa9GjcQkalNxWAU\ni5prAdi252jISUREckvFYBTTa8qYMa2Ml/ceJa6bz0RkClMxOA1rruV4zwAtbfrmMxGZulQMTmOo\nq8jVVSQiU1gsk43M7E7gciAB3OLu61PWrQHuAAaBte5+e7o2ZtYM3AMUA/3Ae939gJldD3wUiANf\nc/evj9cBnqlFZ79WDFZffFbIaUREcuO0ZwZmthJY6O4rgBuBu4ZtchdwLXAlcJWZLRmlzedJftiv\nBB4EPm5mlcBngDXAKuBjZjb9jI9snMysLWdaVQnb9hzVpHUiMmVl0k20GvghgLtvBerMrAbAzOYD\nR9x9j7vHgbXB9unafAR4INhvG1APXAasd/cOd+8GniJZWPJCJBLBmmvpPN7HwfbusOOIiOREJt1E\njcDGlMdtwbLO4GdbyrpWYAEwY6Q27r4NwMyiwM3AbWn2MXu0QHV1FcRi0Qyij6yhoTqr7ZcvbuS5\nra20tPdwvs0a8/NmIttsE0W5sqNc2cvXbIWSK6Mxg2EiY1h3cnlQCO4FHnf3x8zsz7LYPwDt7SdO\nGzKdhoZq2tq6smrTVFcGwMYtB1i+IHc9WGPJNhGUKzvKlb18zTbVco1WQDLpJmoh+df7kCZgf5p1\nc4Jlo7W5B3jZ3T93mn3kjaYZlVSWxXTzmYhMWZkUg0eB6wDMbDnQ4u5dAO6+C6gxs3lmFgOuCbYf\nsU1w1VCfu382Zf/PApeaWa2ZVZEcL3hyXI5unBRFIixqruVwZw+HOjRuICJTz2m7idx9nZltNLN1\nJC/9vNnMbgA63P1B4CbgO8Hm9wfjAtuGtwnW3wyUmdkvgsdb3P0jZnYr8BOSl6F+zt3zbjKgRc21\nPP/yIbbtOcqMaeVhxxERGVcZjRm4+63DFm1KWfcEsCKDNrj7FWn2/wPgB5lkCct5Z9cBsPXVdq5Y\nNur4tojIpKM7kDPUPKuKqvJituxq1/0GIjLlqBhkqCgSYfHcOtq7ejlwZOxXM4mI5CMVgywsPSd5\nWenmnUdCTiIiMr5UDLKwZF5y3GDLrvaQk4iIjC8VgyzMmFbOzLpyXtrdzsBgPOw4IiLjRsUgS0vn\nTaenb5Cd+zvDjiIiMm5UDLK0ZF5y3EBdRSIylagYZGnx3FoiEdi8S4PIIjJ1qBhkqaKsmHNm1/DK\nvk66ewfCjiMiMi5UDMZgybzpxBMJfLcmrhORqUHFYAyWBpeY/nbn4ZCTiIiMDxWDMVgwZxrlpTFe\n3HFYU1OIyJSgYjAGsWgR58+fzqGOHloOHQ87jojIGVMxGKMLF8wAYNMOdRWJyOSnYjBGy+ZPJxKB\nTdsPhR1FROSMqRiMUXVFCQvmTGP7vg6OdfeHHUdE5IyoGJyBCxfUk0jAb15RV5GITG4qBmfg5LiB\nuopEZJJTMTgDcxoqqa8p5bevHNEspiIyqakYnIFIJMIF587gRO8AO/Z1hB1HRGTMVAzO0FBX0Qvq\nKhKRSUzF4AwtnltLWUmUjd6mu5FFZNJSMThDxbEoF547g0MdPew+eCzsOCIiY6JiMA4usQYANnhr\nyElERMYmlslGZnYncDmQAG5x9/Up69YAdwCDwFp3v320Nmb2N8A/AXXufixY1g88lfKUq9198AyP\nbcIsm19PSXER619q5Q/fPJ9IJBJ2JBGRrJy2GJjZSmChu68ws8XA3cCKlE3uAq4G9gG/NLMHgIaR\n2pjZ+4FZQMuwp+lw91VnfDQhKS2OcsGCGWx4qZW9bcdpnlkVdiQRkaxk0k20GvghgLtvBerMrAbA\nzOYDR9x9j7vHgbXB9unaPOjunyZ5tjClnOwqekldRSIy+WTSTdQIbEx53BYs6wx+tqWsawUWADNG\nauPu29I8R5mZ3QfMBR5w96+MFqiuroJYLJpB9JE1NFSPuW06b60p5+4fbeWFHYf58LUXjnk/ucg2\nHpQrO8qVvXzNVii5MhozGGa0DvF0607Xif4J4FskzxieMLMn3H1Duo3b20+cZnfpNTRU09bWNeb2\no1l6znSef/kQL2w9wJwZlVm3z2W2M6Fc2VGu7OVrtqmWa7QCkkk3UQvJM4AhTcD+NOvmBMtGa/M6\n7v5Vdz/m7seBx4DzM8iVdy45byYAG9VVJCKTTCbF4FHgOgAzWw60uHsXgLvvAmrMbJ6ZxYBrgu3T\nthnOku4zs0iwjyuBzWd2WOG46NwZxKJFPLv1oG5AE5FJ5bTdRO6+zsw2mtk6IA7cbGY3kLwC6EHg\nJuA7web3B+MC24a3ATCzTwNvI3nW8GMze9rdP2lme4Dngm0fcvfnxvcwJ0Z5aYyLFiavKtp1oItz\nZteEHUlEJCMZjRm4+63DFm1KWfcEp15qmq4N7v4F4AsjLP9UJjkmgyuWNrLhpVae3nxAxUBEJg3d\ngTzOls2fTlV5Mc9tOchgXNNai8jkoGIwzmLRIn5n8Uw6T/SzeWd72HFERDKiYpADK5YlL6R6evOB\nkJOIiGRGxSAH5s+uYVZdOc9va6O7dyDsOCIip6VikAORSIQVSxvpG4jz621tp28gIhIyFYMcuXzp\nLACe+k3ae+1ERPKGikGOzKyr4Lyza3lp91EOHBn79BkiIhNBxSCHVl40B4AnXhg+Y7eISH5RMcih\n5YsaqCov5le/2U//gO45EJH8pWKQQ8WxIq48v5Fj3f08/7IGkkUkf6kY5NhQV9Evnt8XchIRkfRU\nDHKscboGkkUk/6kYTAANJItIvlMxmADLFzVQXVHMky+20Ns3GHYcEZHXUTGYAMWxIt7yhjkc7xlg\n3W91E5qI5B8VgwnyluVnEYtGeHTDXuL6FjQRyTMqBhNkWmUJly2ZxcEjJ3hxx+Gw44iInELFYAK9\n7ZJmAH66fk/ISURETqViMIHOnlXN4rl1bH21nd0Hu8KOIyJykorBBLvqUp0diEj+UTGYYOcvqGd2\nfQXPbDnIoY7usOOIiAAqBhOuKBLhnSvmMhhP8ONndocdR0QEUDEIxWVLZjGztpwnX2yhvas37Dgi\nIsQy2cjM7gQuBxLALe6+PmXdGuAOYBBY6+63j9bGzP4G+Cegzt2PBcuuBz4KxIGvufvXx+fw8lO0\nqIh3rJjLN378Ej9+9lX+bM2isCOJSIE77ZmBma0EFrr7CuBG4K5hm9wFXAtcCVxlZkvStTGz9wOz\ngJaU/VcCnwHWAKuAj5nZ9DM8rrx3xbJG6mtK+eULLXQc7ws7jogUuEy6iVYDPwRw961AnZnVAJjZ\nfOCIu+9x9ziwNtg+XZsH3f3TJM8WhlwGrHf3DnfvBp4iWVimtFi0iHdcPpf+gTg/eU5jByISrkyK\nQSOQ+s0sbcGykda1ArPTtXH3kS6uT7ePKe+NFzRRV13K4xv3cvSYxg5EJDwZjRkMExnDutHaZL1t\nXV0FsVg0i12eqqGhesxtx9v1v3se//b9TTy6cR8Lz5mRV9lSKVd2lCt7+ZqtUHJlUgxaeO1MAKAJ\n2J9m3ZxgWd8obU63/znAM6MFam8f+5fENDRU09aWP3f/XnhOHY3TK3j0mVd595vnU5pN2Zwg+faa\nDVGu7ORrLsjfbFMt12gFJJNuokeB6wDMbDnQMtTd4+67gBozm2dmMeCaYPu0bUbwLHCpmdWaWRXJ\n8YInM8g1JUSLirh25QLiiQTfXLs17DgiUqBOe2bg7uvMbKOZrSN56efNZnYD0OHuDwI3Ad8JNr/f\n3bcB24a3ATCzTwNvI3km8GMze9rdP2lmtwI/ITmw/Dl37xjfw8xvyxfNYMGcGp7+zX7eelETC+ZM\nCzuSiBSYSGISzq3f1tY15tD5etq3bc9RvvTtX7PwrGncev1yIpH86S/K19dMubKTr7kgf7NNtVwN\nDdVpP1h0B3KeWNRcy2VLG3l5bwfPbj0YdhwRKTAqBnnkQ+9eRixaxPce305370DYcUSkgKgY5JHG\n+krecfnZHD3Wx8PrdoUdR0QKiIpBnnnH5XOZMa2Mn67fQ8uh42HHEZECoWKQZ0qKo/zJ6oUMxhN8\n+6fbmIwD/CIy+agY5KE3LJzB+fPr2fpqO7/6Tbp79URExo+KQR6KRCK8/2qjtCTKdx/bru88EJGc\nUzHIU/XTynjPW86lu3eAbz7ykrqLRCSnVAzy2MqLmjjv7Fo27TjMM1t074GI5I6KQR4rikS44R2L\nKSku4r6fblN3kYjkjIpBnptZW84fv+VcjvcM8LWHNhOPq7tIRMafisEksOoNc1i+qAHfc5QfPb0r\n7DgiMgWpGEwCkUiEG95+HtNrSvmfX+3i5b1Hw44kIlOMisEkUVVezId/bykJEnztoc10negLO5KI\nTCEqBpPIouZa3v3Gczjc2ctX/2czg/F42JFEZIpQMZhkrrliHm9YOIOtr7bz/Z/vCDuOiEwRKgaT\nTFEkwoeuWcLs+goeXb+HpzRdhYiMAxWDSai8NMZfX3sB5aUx/vsRZ/vegvqWUBHJARWDSapxegU3\nvXsp8XiCf/nBJvYf1nTXIjJ2KgaT2LL59Xzg7cbxngG+cv8mjh7THcoiMjYqBpPcmy5o4vffdA6H\nO3v45+9t4kSPvi5TRLKnYjAF/N4V81h1URO7W49x5/de0Pcni0jWVAymgEgkwnuvMlYsncWOlk7u\n/P4mevpUEEQkcyoGU0RRUYQb37mEy5bMYvveDv75+y+qIIhIxmKZbGRmdwKXAwngFndfn7JuDXAH\nMAisdffb07Uxs2bgXiAK7Afe5+69ZtYPPJXylKvdffCMj67AFBVF+NA1ixmMJ9jwUitf/s4LfOw9\nF1JVXhx2NBHJc6c9MzCzlcBCd18B3AjcNWyTu4BrgSuBq8xsyShtbgP+3d3fBGwH/jxY3uHuq1L+\nUyEYo2hREX/5riVcuayRnfs7+eK3NnKksyfsWCKS5zLpJloN/BDA3bcCdWZWA2Bm84Ej7r7H3ePA\n2mD7dG1WAQ8F+30YWDN+hyJDokVFfPCdi7nq0mb2Hz7BF7+1kX1tx8KOJSJ5LJNi0Ai0pTxuC5aN\ntK4VmD1Km0p37x22LUCZmd1nZk+Z2cezOwQZSVEkwh+/9VyuXTmfw529fOHejWzafijsWCKSpzIa\nMxgmMoZ1Iy1PXfYJ4FskxxeeMLMn3H1Duiepq6sgFoueNmg6DQ3VY26ba+Od7YZ3nc+C5un883d/\nzV0PvMgHr1nK769cQCQy2v/G3OcaL8qVnXzNBfmbrVByZVIMWnjtTACgieTg70jr5gTL+tK0OWZm\n5e7enbIt7v7VoQ3N7DHgfCBtMWhvP5FB7JE1NFTT1tY15va5lKts551Vw6euX85dD7zI3Q9vZvOO\nQ7z/aqO8NLO/BfL1NVOu7ORrLsjfbFMt12gFJJNuokeB6wDMbDnQ4u5dAO6+C6gxs3lmFgOuCbZP\n1+ZnJAebCX4+Ykn3mVkk2MeVwOasj1JGdc7sGj7zgUtZMKeGZ7cc5Lb/3sDeVo0jiEjSaYuBu68D\nNprZOpJXBd1sZjeY2R8Em9wEfAd4Erjf3beN1CbY9rPAB8zsSWA68N/u7sAe4DmSl5eudffnxu8Q\nZUhddSmf+rPlXP07zRw8coLbv7mBxzbuJZ5IhB1NREIWSUzCD4K2tq4xh87X0z6Y2GzPv9zG3T/a\nyvGeAZbMq+ODb19M/bSy0HNlQ7myk6+5IH+zTbVcDQ3VaQcLdQdygXrDwgZu/9BlXLCgni272vnM\n3c/yixf26SxBpECpGBSw2qpSbrnuAj749vNIJOCbjzhfvHcjuw/m319CIpJbKgYFLhKJ8KYLm/jC\nX1zOpefNZEdLJ7d9YwPfetTpPN4XdjwRmSAqBgIkB5dv+v1lfPw9FzKjtozHf72PT/3X0zz81E56\nNCW2yJSnYiCnWDa/ns9/6DKuf9siSmJFPPjkTv7ySz/jFy/sY2AwHnY8EcmRsdyBLFNcLFrE6ovP\n4opljfz42d38dMMevvmI8/BTu7j60mbefFETZSV664hMJTozkLTKS2P84Zvn81+3ruaqS5s50TPA\ndx/fzt/+xzoefOIVjSmITCH6805Oq35aOX+yeiHXXDGPx3+9l59t2MvD63bx42dfZfmiBlZdNAc7\nuzbr+Y5EJH+oGEjGqsqLedeV53D1pWfzq9/s5+fP7+O5ra08t7WVxukVrLyoicuXzGJaVWnYUUUk\nSyoGkrXSkiirLz6Lty6fw8t7O/jFC/vY8FIb9z++ne/9fDvnnV3HZUtmcbE1UFmmb1kTmQxUDGTM\nIpEIi5prWdRcy5+u7uOZLQd5bstBtr7aztZX27n3J87Sc6Zz0bkzuPDcGdRV64xBJF+pGMi4qK4o\n4W2XNPO2S5ppO9rNc1sP8uyWVl7ccZgXdxyGnzhnz6rionNnsPSc6Zwzu4ZYVNcviOQLFQMZdw21\n5bxzxTzeuWIerUe7eXH7ITZtP8RLu4+y++AxHnpqF6XFURaeNY3Fc+uws+s4e1aVioNIiFQMJKdm\n1paz5pJm1lzSTHfvAFt2tfPSq+1s3d3Ob3ce4bc7jwDJexvmNlYxf/Y05jfVsKCphvppZbpCSWSC\nqBjIhCkvjXGxNXCxNQDQcayXl3YfZdueo7zS0snOli527Os8uX11RTHNM6s4q6GKOQ2VNM+soqm+\nkpLisX/lqYiMTMVAQjOtqpTLlszisiWzAOjtH+TVA1280tLJKy0d7DrQxZZd7WzZ1X6yTSQCM+sq\naKwrZ+6caVSXxphVV87MunLqp5URLVJXk8hYqBhI3igtjp68OmlId+8A+w4dZ2/rMfa2HWNv23H2\ntR1j05ETbNpx+JT20aII9dPKaKgtp666lOnVpUyvKaOuuvTk4/LSmLqeREagYiB5rbw0xrlzpnHu\nnGmnLD/W3U8/EXznIVrbuzl4pJvWoydobe9mczAOMZLSkijTKkqoriimOvhZU1lCdXkx1ZXB8vIS\nqsqLKS+NUVYapUjFQwqAioFMSlXlxTQ0VFNX/vq3cG/fIEe6emjv6qW9q5cjwc/2zh6OdPXSeaKP\nwwd6GIyf/lvdIiQLUnlpjIqy4Oewx2UlUUpiRZQWRyktidLQdpye472UlESTy4qjlBRHKS0uoqRY\nxUXyk4qBTDmlJVFm11cyu74y7TaJRIITvQN0Hu+j60R/8F8fXSf66DzRz4mefk70DNDdO8CJ4L9D\nHd109w6ecb5YNEIsWhT8l/y9OFZ0cllxNEL0lGURiqNFyWXRIiJFyS6xoqJI8mckkvK4iKIIr61L\n2a52WgfHj/eeuj4SIRKJEIkkC9/J34f9HCpgRanLYcRtI5FIcgbMCKfuP2hD8FwEzwdQdqKP4z39\nBKuH1gbbnPr6DT2OEEndLGWf6dufvm3hFmoVAylIkUiEyrJiKsuKmV2febt4PEFP3wAnepIFort3\ngN7+QXr74/T2DdI3MEhxSYwjR7uD5YP0Bev7gt/7B+L0D8YZGEwwEPze293PQMoyfRN1fohEgAQp\nRWP0ApVmL+n3nVULiEaLuPGdi7m6oXq0JxwTFQORLBQVRagoK6ZilDmXGhqqaWsb+/dIJxIJBuOJ\nk8WhfyAe/J58HI8niCeSPwfjwc9hj4e2Obk+nqCispSOju6Tj4eeJxE8ZyKR8pOhx68tiyeSJSqe\nui2p25zaJh7sh8Tr2yQPlJO/l5RE6U35Rr3EsGqYCBYML5JD253c66k/Tm6QIP0+X7+v1xYUl8To\n6xsYtnzk5xxJulXDs2TWCqJFRdRUlozWeMxUDETyTCQSOdl9NJ7OtEjlUr5my9dcuaCLskVEJLMz\nAzO7E7ic5PnLLe6+PmXdGuAOYBBY6+63p2tjZs3AvUAU2A+8z917zex64KNAHPiau399vA5QRERO\n77RnBma2Eljo7iuAG4G7hm1yF3AtcCVwlZktGaXNbcC/u/ubgO3An5tZJfAZYA2wCviYmU0/4yMT\nEZGMZdJNtBr4IYC7bwXqzKyJR/V7AAAF50lEQVQGwMzmA0fcfY+7x4G1wfbp2qwCHgr2+zDJAnAZ\nsN7dO9y9G3iKZGEREZEJkkkxaATaUh63BctGWtcKzB6lTaW7955m26HlIiIyQcZyNdFoV9SmWzfS\n8my2PUVdXQWx2NhnrmzIwTW64yVfsylXdpQre/marVByZVIMWnjtTACgieTg70jr5gTL+tK0OWZm\n5UF30NC2I+3jmdECtbefyCD2yPL5UrF8zaZc2VGu7OVrtqmWa7QCkkk30aPAdQBmthxocfcuAHff\nBdSY2TwziwHXBNuna/MzkoPNBD8fAZ4FLjWzWjOrIjle8GSWxygiImcgMvwuvJGY2ZeAN5O89PNm\n4A1Ah7s/aGZvBv4+2PQBd//Hkdq4+yYzmw18EygDXgU+6O79ZnYd8LckL0P9V3f/9ngepIiIjC6j\nYiAiIlOb7kAWEREVAxERUTEQERFUDEREBBUDERFBxUBERCiwL7cZbSruELL8A/Amkv8Pvgi8C7gY\nOBxs8mV3/1EIuVYB3wc2B4t+A/wDI0w9PsG5bgTel7LoEmADUAkcD5b9L3ffOIGZlgH/A9zp7v+W\nL1O0p8l1D1AM9APvdfcDZtZPcmLIIavd/cy/5DnzXN9ghPd8Hrxe3wcagtXTSc6IcAfJfwtD7682\nd/+jHOca/hmxnhy+vwqmGKROq21mi4G7gRUhZXkLsCzIUg88DzwO/G93/39hZBrml+5+3dADM7uH\n5NTj3zezO4A/B/5zIgMFb/CvB3lWAu8BlpK8cfG3E5klyFAJ/CvwWMrioSnaT75OZvZNklO0/w7J\naVrWm9mD7n5kAnN9nuSHxPfM7Gbg48AnSd44uioXOTLMBcPe8ylT2of2eqV+yJvZ3cD/fW3VhL1e\nI31GPEYO31+F1E2UdiruEDwBDL3hjpL863bsM+/l3ipeP/V4mD4D3B5yhl7gHSTn1hqyivCnaB8p\n10eAB4Lf24D6HD5/OiPlGkk+vF4AmJkBte7+XA6fP52RPiNWkcP3V8GcGZCcDC+1C2FoWu3OiQ4S\nnIoPdW3cSPJ7IAaBvzKzj5Ocxvuv3P3QRGcLLDGzh0ieIn+OkaceD4WZXQrsCbo5AG4zsxnAVuCj\nwT+InHP3AWAgyDAk9CnaR8rl7scBzCxKcjqZ24JVZWZ2HzCX5FQyX5nIXIFT3vPkweuV4haSZw1D\nGs3sByQn3vz3XE6bk+Yz4upcvr8K6cxguNNOlZ1rZvZukv+j/4pkX+Ct7v5W4AXg70KK9TLJAvBu\n4AMku2ZS/2gI+3X7EPCN4Pd/Af7W3VPnzcoXY56iPReCQnAv8Li7D3WJfAL4MHAVcL2ZXTLBsTJ5\nz4f1epUAb3T3nweLDgP/B/hTkuN7twdzreU6R+pnRKpxf38V0pnBaFNxTzgzuxr4NPC77t7BqX2p\nDzHBffJD3H0fcH/wcIeZHSA5q+zwqcfDsgr4awB3fzBl+cPAH4cRKMW4TNGeI/cAL7v754YWuPtX\nh343s8eA80kOyk+IlKIEr73nf0B+vF4rgZPdQ8Gsy/cEDw+Z2QbgPHL4GTL8M8LMcvr+KqQzg7RT\ncU80M5sGfBm4Zmigx8weCL5GFJIfeBM+KBrkuN7MPhH83gjMIvmPYPjU42FkawKOuXufmUXM7Gdm\nVhusXkVIr1mKvJyiPbjapM/dP5uyzMzsvuB1jAW5NqfdSW5yjfSeD/31ClwKbBp6YGZvMbOvBL9X\nAhcB23L15CN9RpDj91dBzVo60rTaIeX4MMlT4tQ30z0kTwVPAMdIXiXTGkK2auA+oBYoIdll9Dwj\nTD0eQraLgc+7+9uDx+8BPkWyb3UfcKO7j/2bj7LP8k/APJKXa+4DrifZhRXaFO1pcs0EenhtfGyL\nu3/EzP4eeCvJfw8PufsXJjjXvwK3Muw9nwev1x+SfN//yt3vD7aLkbyqyEhe7PGf7n7PSPscp1wj\nfUZ8IMiQk/dXQRUDEREZWSF1E4mISBoqBiIiomIgIiIqBiIigoqBiIigYiAiIqgYiIgI8P8B0PnL\n/mqLXbgAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X9uiFJ-ogBRi",
        "colab_type": "text"
      },
      "source": [
        "### Loading a Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Doot7_0fY5tF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "unlabeled_images_test = pd.read_csv('gdrive/My Drive/dataML/test.csv')\n",
        "#unlabeled_images_test = pd.read_csv('test.csv')\n",
        "\n",
        "X_unlabeled = unlabeled_images_test.values.reshape(unlabeled_images_test.shape[0],28,28,1)/255"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dbtjpi8JgBRj",
        "colab_type": "code",
        "outputId": "ee65fc89-ef7e-4b99-b2bb-4d24d4e518c3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "with tf.Session() as sess:\n",
        "    \n",
        "    # Restore the model\n",
        "    saver.restore(sess, 'models_saving/my_model.ckpt')\n",
        "    \n",
        "\n",
        "    # Fetch Back Results\n",
        "    label = sess.run(Y, feed_dict={X:X_unlabeled,drop_rate:0})"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from models_saving/my_model.ckpt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MZ1xS7oJZYkV",
        "colab_type": "code",
        "outputId": "6d483b80-a575-49e8-ce54-db9c91392100",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "label"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([2, 0, 9, ..., 3, 9, 2])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ojRKNc76gBRx",
        "colab_type": "text"
      },
      "source": [
        "## Predict the unlabeled test sets using the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yQrGiou8gBRy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "imageId = np.arange(1,label.shape[0]+1).tolist()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9Oj02pY3gBR1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "prediction_pd = pd.DataFrame({'ImageId':imageId, 'Label':label})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VLjMgeXEgBR4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "prediction_pd.to_csv('gdrive/My Drive/dataML/out_cnn4.csv',sep = ',', index = False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ivz_HMLnZ684",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}